Started by user Ritesh Agarwal [EnvInject] - Loading node environment variables. Building remotely on magma-sc1403-32g-24c (magma_slave) in workspace /data/workspace/magma_temp_job3 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins6576394760968014429.sh ++ echo magma + desc=magma + echo Desc: 7.1.0-1817 - magma Desc: 7.1.0-1817 - magma Success build forhudson.tasks.Shell@317e8f66 [description-setter] Description set: 7.1.0-1817 - magma Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@57cd7ebc Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/magma_temp_job3 # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision b02f0a6891dcb53fcebd2cfe350412947f1e06c5 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f b02f0a6891dcb53fcebd2cfe350412947f1e06c5 > /usr/bin/git rev-list 1d48d12bd1f657c3a7b5f59832f5f09e52604af4 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #256 jenkins-magma_temp_job3-256 # timeout=10 No emails were triggered. [magma_temp_job3] $ /bin/bash /tmp/jenkins299557271058878722.sh [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 data_path:/data [servers] 1:172.23.110.64 2:172.23.110.65 3:172.23.110.66 4:172.23.110.67 5:172.23.110.68 [membase] rest_username:Administrator rest_password:password [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins2048925240512361778.sh + echo '' + echo '##############################################' ############################################## + echo 'Installing dependencies from requirements.txt' Installing dependencies from requirements.txt + echo '##############################################' ############################################## + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + echo '#################### Done ####################' #################### Done #################### + echo '' + echo version=7.1.0-1817 version=7.1.0-1817 + mkdir tr_for_install + cd tr_for_install + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... + cd testrunner + git checkout master Already on 'master' + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + py_executable=python3 + python3 scripts/new_install.py -i /tmp/magma_temp_job3.ini -p timeout=1200,get-cbcollect-info=True,version=7.1.0-1817,product=cb,debug_logs=False,url=,install_debug_info=true 2021-12-01 11:07:46,700 - root - WARNING - URL: is not valid, will use version to locate build 2021-12-01 11:07:46,701 - root - INFO - SSH Connecting to 172.23.110.64 with username:root, attempt#1 of 5 2021-12-01 11:07:46,854 - root - INFO - SSH Connected to 172.23.110.64 as root 2021-12-01 11:07:47,201 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:47,609 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:47,611 - root - INFO - SSH Connecting to 172.23.110.65 with username:root, attempt#1 of 5 2021-12-01 11:07:47,787 - root - INFO - SSH Connected to 172.23.110.65 as root 2021-12-01 11:07:48,160 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:48,542 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:48,544 - root - INFO - SSH Connecting to 172.23.110.66 with username:root, attempt#1 of 5 2021-12-01 11:07:48,699 - root - INFO - SSH Connected to 172.23.110.66 as root 2021-12-01 11:07:49,070 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:49,455 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:49,457 - root - INFO - SSH Connecting to 172.23.110.67 with username:root, attempt#1 of 5 2021-12-01 11:07:49,615 - root - INFO - SSH Connected to 172.23.110.67 as root 2021-12-01 11:07:50,025 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:50,413 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:50,414 - root - INFO - SSH Connecting to 172.23.110.68 with username:root, attempt#1 of 5 2021-12-01 11:07:50,591 - root - INFO - SSH Connected to 172.23.110.68 as root 2021-12-01 11:07:50,986 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:51,366 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:51,369 - root - INFO - SSH Connecting to 172.23.110.64 with username:root, attempt#1 of 5 2021-12-01 11:07:51,517 - root - INFO - SSH Connected to 172.23.110.64 as root 2021-12-01 11:07:51,866 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:52,272 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:52,275 - root - INFO - SSH Connecting to 172.23.110.65 with username:root, attempt#1 of 5 2021-12-01 11:07:52,438 - root - INFO - SSH Connected to 172.23.110.65 as root 2021-12-01 11:07:52,785 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:53,165 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:53,167 - root - INFO - SSH Connecting to 172.23.110.66 with username:root, attempt#1 of 5 2021-12-01 11:07:53,311 - root - INFO - SSH Connected to 172.23.110.66 as root 2021-12-01 11:07:53,665 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:54,053 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:54,055 - root - INFO - SSH Connecting to 172.23.110.67 with username:root, attempt#1 of 5 2021-12-01 11:07:54,202 - root - INFO - SSH Connected to 172.23.110.67 as root 2021-12-01 11:07:54,575 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:54,955 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:54,956 - root - INFO - SSH Connecting to 172.23.110.68 with username:root, attempt#1 of 5 2021-12-01 11:07:55,106 - root - INFO - SSH Connected to 172.23.110.68 as root 2021-12-01 11:07:55,465 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:55,843 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:55,844 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,881 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,881 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,901 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,901 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,902 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,902 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,903 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,904 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,905 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,906 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,907 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,908 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,909 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,909 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,911 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,911 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,912 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,912 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,914 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:08:02,053 - root - INFO - Done with uninstall on 172.23.110.67. 2021-12-01 11:08:02,420 - root - INFO - Done with uninstall on 172.23.110.66. 2021-12-01 11:09:29,457 - root - INFO - Done with uninstall on 172.23.110.64. 2021-12-01 11:09:38,934 - root - INFO - Done with uninstall on 172.23.110.65. 2021-12-01 11:09:39,975 - root - INFO - Done with uninstall on 172.23.110.68. 2021-12-01 11:09:42,559 - root - INFO - running command.raw on 172.23.110.64: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:42,588 - root - INFO - command executed successfully with root 2021-12-01 11:09:44,796 - root - INFO - running command.raw on 172.23.110.64: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:44,824 - root - INFO - command executed successfully with root 2021-12-01 11:09:45,651 - root - INFO - running command.raw on 172.23.110.65: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:45,676 - root - INFO - command executed successfully with root 2021-12-01 11:09:47,711 - root - INFO - running command.raw on 172.23.110.65: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:47,736 - root - INFO - command executed successfully with root 2021-12-01 11:09:48,498 - root - INFO - running command.raw on 172.23.110.66: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:48,524 - root - INFO - command executed successfully with root 2021-12-01 11:09:49,196 - root - INFO - running command.raw on 172.23.110.66: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:49,222 - root - INFO - command executed successfully with root 2021-12-01 11:09:49,906 - root - INFO - running command.raw on 172.23.110.67: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:49,932 - root - INFO - command executed successfully with root 2021-12-01 11:09:50,744 - root - INFO - running command.raw on 172.23.110.67: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:50,769 - root - INFO - command executed successfully with root 2021-12-01 11:09:51,602 - root - INFO - running command.raw on 172.23.110.68: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:51,628 - root - INFO - command executed successfully with root 2021-12-01 11:09:52,319 - root - INFO - running command.raw on 172.23.110.68: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:52,344 - root - INFO - command executed successfully with root 2021-12-01 11:11:26,097 - root - INFO - Done with install on 172.23.110.64. 2021-12-01 11:11:26,097 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:26,106 - root - INFO - running command.raw on 172.23.110.64: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.64 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:26,527 - root - INFO - Done with install on 172.23.110.65. 2021-12-01 11:11:26,527 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:26,533 - root - INFO - command executed successfully with root 2021-12-01 11:11:26,536 - root - INFO - running command.raw on 172.23.110.65: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.65 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:26,538 - root - INFO - running command.raw on 172.23.110.64: rm -rf /data/* 2021-12-01 11:11:27,002 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,009 - root - INFO - running command.raw on 172.23.110.65: rm -rf /data/* 2021-12-01 11:11:27,160 - root - INFO - Done with install on 172.23.110.66. 2021-12-01 11:11:27,160 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:27,168 - root - INFO - running command.raw on 172.23.110.66: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.66 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:27,272 - root - INFO - Done with install on 172.23.110.67. 2021-12-01 11:11:27,273 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:27,282 - root - INFO - running command.raw on 172.23.110.67: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.67 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:27,639 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,644 - root - INFO - running command.raw on 172.23.110.66: rm -rf /data/* 2021-12-01 11:11:27,732 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,736 - root - INFO - running command.raw on 172.23.110.67: rm -rf /data/* 2021-12-01 11:11:27,788 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,788 - root - INFO - running command.raw on 172.23.110.66: chown -R couchbase:couchbase /data 2021-12-01 11:11:27,853 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,853 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:11:28,306 - root - INFO - command executed successfully with root 2021-12-01 11:11:28,306 - root - INFO - running command.raw on 172.23.110.67: chown -R couchbase:couchbase /data 2021-12-01 11:11:28,412 - root - INFO - command executed successfully with root 2021-12-01 11:11:28,412 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:11:30,610 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:11:31,421 - root - INFO - Done with install on 172.23.110.68. 2021-12-01 11:11:31,421 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:31,431 - root - INFO - running command.raw on 172.23.110.68: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.68 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:31,614 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.66 2021-12-01 11:11:31,614 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:11:31,619 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:11:31,620 - root - INFO - /node/controller/setupServices params on 172.23.110.66: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:11:31,653 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:11:31,653 - root - INFO - settings/web params on 172.23.110.66:8091:port=8091&username=Administrator&password=password 2021-12-01 11:11:31,755 - root - INFO - --> status:True 2021-12-01 11:11:31,755 - root - INFO - Done with init on 172.23.110.66. 2021-12-01 11:11:31,781 - root - INFO - Done with cleanup on 172.23.110.66. 2021-12-01 11:11:31,902 - root - INFO - command executed successfully with root 2021-12-01 11:11:31,907 - root - INFO - running command.raw on 172.23.110.68: rm -rf /data/* 2021-12-01 11:11:32,151 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:11:33,157 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.67 2021-12-01 11:11:33,157 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:11:33,162 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:11:33,162 - root - INFO - /node/controller/setupServices params on 172.23.110.67: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:11:33,195 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:11:33,196 - root - INFO - settings/web params on 172.23.110.67:8091:port=8091&username=Administrator&password=password 2021-12-01 11:11:33,290 - root - INFO - --> status:True 2021-12-01 11:11:33,290 - root - INFO - Done with init on 172.23.110.67. 2021-12-01 11:11:33,487 - root - INFO - Done with cleanup on 172.23.110.67. 2021-12-01 11:12:58,829 - root - INFO - command executed successfully with root 2021-12-01 11:12:58,830 - root - INFO - running command.raw on 172.23.110.65: chown -R couchbase:couchbase /data 2021-12-01 11:12:58,856 - root - INFO - command executed successfully with root 2021-12-01 11:12:58,856 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:13:01,648 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:13:02,653 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.65 2021-12-01 11:13:02,653 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:13:02,658 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:13:02,658 - root - INFO - /node/controller/setupServices params on 172.23.110.65: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:13:02,691 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:13:02,691 - root - INFO - settings/web params on 172.23.110.65:8091:port=8091&username=Administrator&password=password 2021-12-01 11:13:02,773 - root - INFO - --> status:True 2021-12-01 11:13:02,773 - root - INFO - Done with init on 172.23.110.65. 2021-12-01 11:13:02,801 - root - INFO - Done with cleanup on 172.23.110.65. 2021-12-01 11:13:11,349 - root - INFO - command executed successfully with root 2021-12-01 11:13:11,349 - root - INFO - running command.raw on 172.23.110.68: chown -R couchbase:couchbase /data 2021-12-01 11:13:11,379 - root - INFO - command executed successfully with root 2021-12-01 11:13:11,379 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:13:14,344 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:13:15,359 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.68 2021-12-01 11:13:15,359 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:13:15,365 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:13:15,365 - root - INFO - /node/controller/setupServices params on 172.23.110.68: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:13:15,400 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:13:15,400 - root - INFO - settings/web params on 172.23.110.68:8091:port=8091&username=Administrator&password=password 2021-12-01 11:13:15,499 - root - INFO - --> status:True 2021-12-01 11:13:15,499 - root - INFO - Done with init on 172.23.110.68. 2021-12-01 11:13:15,526 - root - INFO - Done with cleanup on 172.23.110.68. 2021-12-01 11:13:23,470 - root - INFO - command executed successfully with root 2021-12-01 11:13:23,471 - root - INFO - running command.raw on 172.23.110.64: chown -R couchbase:couchbase /data 2021-12-01 11:13:23,501 - root - INFO - command executed successfully with root 2021-12-01 11:13:23,501 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:13:26,384 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:13:27,391 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.64 2021-12-01 11:13:27,391 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:13:27,396 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:13:27,396 - root - INFO - /node/controller/setupServices params on 172.23.110.64: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:13:27,430 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:13:27,430 - root - INFO - settings/web params on 172.23.110.64:8091:port=8091&username=Administrator&password=password 2021-12-01 11:13:27,528 - root - INFO - --> status:True 2021-12-01 11:13:27,528 - root - INFO - Done with init on 172.23.110.64. 2021-12-01 11:13:27,561 - root - INFO - Done with cleanup on 172.23.110.64. 2021-12-01 11:13:27,583 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,619 - root - INFO - cluster:C1 node:172.23.110.64:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C2 node:172.23.110.65:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C3 node:172.23.110.66:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C4 node:172.23.110.67:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C5 node:172.23.110.68:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.64 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.65 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.66 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.67 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.68 2021-12-01 11:13:27,620 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - TOTAL INSTALL TIME = 341 seconds + status=0 + cd ../.. + rm -rf tr_for_install + git clone https://github.com/gradle/guides.git Cloning into 'guides'... [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins8282771302773184452.sh + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest' Starting a Gradle Daemon, 33 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.2.1 Using Java_client :: 3.2.2 Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/ssh.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/eagles_all_around.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main scripts/install.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest > Task :compileJava warning: unknown enum constant javax.annotation.meta.When.MAYBE reason: class file for javax.annotation.meta.When not found Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 warning > Task :testrunner Global Test input params: {'bucket_eviction_policy': 'fullEviction', 'bucket_storage': 'magma', 'cluster_name': 'magma_temp_job3', 'ini': '/tmp/magma_temp_job3.ini', 'num_nodes': 5, 'rerun': 'False'} Logs will be stored at /data/workspace/magma_temp_job3/logs/testrunner-21-Dec-01_11-13-54/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey' Test Input params: {'fts_nodes': '0', 'doc_size': '1536', 'ops_rate': '200000', 'key_type': 'RandomKey', 'bucket_type': 'membase', 'num_collections': '50', 'maxttl': '10', 'graceful': 'True', 'rerun': 'False', 'num_nodes': 5, 'doc_ops': 'create:update:delete:read', 'ramQuota': '102400', 'eviction_policy': 'fullEviction', 'sdk_timeout': '60', 'case_number': 1, 'num_indexes': '100', 'cluster_name': 'magma_temp_job3', 'batch_size': '1000', 'ini': '/tmp/magma_temp_job3.ini', 'num_buckets': '1', 'log_level': 'debug', 'bucket_storage': 'magma', 'rebl_ops_rate': '50000', 'bucket_eviction_policy': 'fullEviction', 'iterations': '5', 'logs_folder': '/data/workspace/magma_temp_job3/logs/testrunner-21-Dec-01_11-13-54/test_1', 'skip_cleanup': 'True', 'nodes_init': '3', 'pc': '10', 'num_items': '40000000', 'cbas_nodes': '0', 'assert_crashes_on_load': 'True', 'mutation_perc': '100', 'bucket_names': 'GleamBook', 'randomize_doc_size': 'False', 'index_nodes': '0', 'randomize_value': 'True', 'infra_log_level': 'debug', 'key_size': '18'} SystemTestMagma (aGoodDoctor.Hospital.Murphy) ... 2021-12-01 11:14:00,973 | infra | INFO | MainThread | [task_manager:__init__:13] Initiating TaskManager with 30 threads 2021-12-01 11:14:00,977 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2021-12-01 11:14:00,979 | test | INFO | MainThread | [basetestcase:log_setup_status:658] ========= Murphy setup started for test #1 SystemTestMagma ========= 2021-12-01 11:14:01,197 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:01,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:01,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:01,648 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:01,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:01,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:01,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:01,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:01,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:01,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:01,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:01,801 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:14:01,802 | test | INFO | MainThread | [basetestcase:log_setup_status:658] ========= BaseTestCase setup started for test #1 SystemTestMagma ========= 2021-12-01 11:14:01,805 | test | DEBUG | MainThread | [basetestcase:setUp:327] Cleaning up cluster 2021-12-01 11:14:02,200 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 120secs 2021-12-01 11:14:02,288 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-01 11:14:02,289 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:02,305 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-01 11:14:02,308 | test | WARNING | MainThread | [basetestcase:tearDownEverything:583] CLEANUP WAS SKIPPED 2021-12-01 11:14:02,309 | infra | INFO | MainThread | [basetestcase:tearDownEverything:626] ========== tasks in thread pool ========== 2021-12-01 11:14:02,311 | infra | INFO | MainThread | [basetestcase:tearDownEverything:628] ========================================== 2021-12-01 11:14:02,312 | test | INFO | MainThread | [basetestcase:initialize_cluster:414] Initializing cluster : C1 2021-12-01 11:14:02,316 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:02,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:02,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:02,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:02,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:02,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:02,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:02,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:02,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:02,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:02,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:02,990 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:03,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:03,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:03,450 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:03,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:03,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:03,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:03,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:03,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:03,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:03,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:03,641 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.64 - Checking if couchbase-server is installed 2021-12-01 11:14:03,673 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.64 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:03,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:03,746 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.64 - Running systemd command 2021-12-01 11:14:03,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: systemctl stop couchbase-server.service 2021-12-01 11:14:05,739 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.64 port:8091 ssh_username:root 2021-12-01 11:14:05,740 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:14:05,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /data/* 2021-12-01 11:14:05,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:05,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:05,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:05,855 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:05,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:06,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:06,322 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:06,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:06,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:06,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:06,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:06,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:06,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:06,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:06,476 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.64 - Checking if couchbase-server is installed 2021-12-01 11:14:06,503 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.64 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:06,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:06,572 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.64 - Running systemd command 2021-12-01 11:14:06,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: systemctl start couchbase-server.service 2021-12-01 11:14:06,604 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.64 port:8091 ssh_username:root 2021-12-01 11:14:06,605 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0x9 type=client open_count=1 channel=[id: 0x72876e8d, 0.0.0.0/0.0.0.0:49257] timeout=300.0>) 2021-12-01 11:14:06,615 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:07,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:07,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xa type=client open_count=1 channel=[id: 0xf1568f25, 0.0.0.0/0.0.0.0:37481] timeout=300.0>) 2021-12-01 11:14:09,161 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:09,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:09,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0x0421a346, 0.0.0.0/0.0.0.0:54683] timeout=300.0>) 2021-12-01 11:14:11,197 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:11,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:11,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0xcb5b38ab, 0.0.0.0/0.0.0.0:58711] timeout=300.0>) 2021-12-01 11:14:13,220 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:13,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:13,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:15,255 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:15,267 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:14:15,273 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:14:15,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:15,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:14:15,752 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:15,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:14:15,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:14:15,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:14:15,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:14:15,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:14:15,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:14:15,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:14:15,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:14:16,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:16,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:14:16,355 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:16,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:14:16,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:14:16,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:14:16,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:14:16,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:14:16,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:14:16,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:14:16,456 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.65 - Checking if couchbase-server is installed 2021-12-01 11:14:16,471 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.65 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:16,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:16,517 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.65 - Running systemd command 2021-12-01 11:14:16,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: systemctl stop couchbase-server.service 2021-12-01 11:14:18,553 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.65 port:8091 ssh_username:root 2021-12-01 11:14:18,555 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:14:18,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /data/* 2021-12-01 11:14:18,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:18,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:18,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:18,651 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:14:18,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:19,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:14:19,121 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:19,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:14:19,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:14:19,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:14:19,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:14:19,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:14:19,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:14:19,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:14:19,276 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.65 - Checking if couchbase-server is installed 2021-12-01 11:14:19,303 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.65 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:19,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:19,375 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.65 - Running systemd command 2021-12-01 11:14:19,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: systemctl start couchbase-server.service 2021-12-01 11:14:19,411 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.65 port:8091 ssh_username:root 2021-12-01 11:14:19,411 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xa1630016, 0.0.0.0/0.0.0.0:43236] timeout=300.0>) 2021-12-01 11:14:19,417 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:19,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:19,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0xda290cfa, 0.0.0.0/0.0.0.0:41773] timeout=300.0>) 2021-12-01 11:14:21,444 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:21,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:21,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x3d54f4f5, 0.0.0.0/0.0.0.0:45356] timeout=300.0>) 2021-12-01 11:14:23,467 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:23,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:23,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x19478796, 0.0.0.0/0.0.0.0:51331] timeout=300.0>) 2021-12-01 11:14:25,484 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:25,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:25,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xc2fe3ab1, 0.0.0.0/0.0.0.0:34107] timeout=300.0>) 2021-12-01 11:14:27,500 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:27,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:27,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:29,530 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:29,543 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:14:29,546 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:14:29,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:29,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:14:30,026 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:30,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:14:30,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:14:30,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:14:30,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:14:30,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:14:30,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:14:30,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:14:30,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:14:30,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:30,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:14:30,671 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:30,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:14:30,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:14:30,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:14:30,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:14:30,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:14:30,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:14:30,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:14:30,806 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.66 - Checking if couchbase-server is installed 2021-12-01 11:14:30,828 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.66 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:30,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:30,888 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.66 - Running systemd command 2021-12-01 11:14:30,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: systemctl stop couchbase-server.service 2021-12-01 11:14:33,275 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.66 port:8091 ssh_username:root 2021-12-01 11:14:33,276 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:14:33,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /data/* 2021-12-01 11:14:33,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:33,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:33,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:33,371 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:14:33,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:33,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:14:33,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:33,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:14:33,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:14:33,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:14:33,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:14:33,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:14:33,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:14:33,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:14:33,966 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.66 - Checking if couchbase-server is installed 2021-12-01 11:14:33,987 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.66 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:34,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:34,043 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.66 - Running systemd command 2021-12-01 11:14:34,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: systemctl start couchbase-server.service 2021-12-01 11:14:34,075 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.66 port:8091 ssh_username:root 2021-12-01 11:14:34,075 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xbe5cf474, 0.0.0.0/0.0.0.0:35431] timeout=300.0>) 2021-12-01 11:14:34,081 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:34,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:34,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x151a582c, 0.0.0.0/0.0.0.0:52202] timeout=300.0>) 2021-12-01 11:14:36,098 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:36,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:36,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xbb3d648c, 0.0.0.0/0.0.0.0:59422] timeout=300.0>) 2021-12-01 11:14:38,111 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:38,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:38,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0xec8a722e, 0.0.0.0/0.0.0.0:34497] timeout=300.0>) 2021-12-01 11:14:40,128 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:40,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:40,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:42,266 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:42,279 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:14:42,283 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:14:42,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:42,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:14:42,801 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:42,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:14:42,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:14:42,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:14:42,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:14:42,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:14:42,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:14:42,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:14:42,999 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:14:43,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:43,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:14:43,482 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:43,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:14:43,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:14:43,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:14:43,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:14:43,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:14:43,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:14:43,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:14:43,644 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.67 - Checking if couchbase-server is installed 2021-12-01 11:14:43,673 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.67 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:43,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:43,746 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.67 - Running systemd command 2021-12-01 11:14:43,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: systemctl stop couchbase-server.service 2021-12-01 11:14:45,862 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.67 port:8091 ssh_username:root 2021-12-01 11:14:45,864 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:14:45,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /data/* 2021-12-01 11:14:45,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:45,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:45,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:45,960 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:14:46,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:46,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:14:46,461 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:46,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:14:46,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:14:46,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:14:46,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:14:46,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:14:46,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:14:46,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:14:46,627 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.67 - Checking if couchbase-server is installed 2021-12-01 11:14:46,654 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.67 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:46,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:46,727 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.67 - Running systemd command 2021-12-01 11:14:46,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: systemctl start couchbase-server.service 2021-12-01 11:14:46,765 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.67 port:8091 ssh_username:root 2021-12-01 11:14:46,765 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x03727303, 0.0.0.0/0.0.0.0:41379] timeout=300.0>) 2021-12-01 11:14:46,769 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:46,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:46,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x64fa0c93, 0.0.0.0/0.0.0.0:58259] timeout=300.0>) 2021-12-01 11:14:48,786 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:48,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:48,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x4dd60e09, 0.0.0.0/0.0.0.0:33921] timeout=300.0>) 2021-12-01 11:14:50,802 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:50,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:50,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x7610344e, 0.0.0.0/0.0.0.0:39587] timeout=300.0>) 2021-12-01 11:14:52,821 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:52,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:52,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:55,150 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:55,163 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:14:55,165 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:14:55,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:55,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:14:55,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:55,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:14:55,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:14:55,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:14:55,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:14:55,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:14:55,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:14:55,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:14:55,857 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:14:55,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:56,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:14:56,323 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:56,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:14:56,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:14:56,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:14:56,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:14:56,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:14:56,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:14:56,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:14:56,460 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.68 - Checking if couchbase-server is installed 2021-12-01 11:14:56,480 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.68 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:56,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:56,536 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.68 - Running systemd command 2021-12-01 11:14:56,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: systemctl stop couchbase-server.service 2021-12-01 11:14:58,479 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.68 port:8091 ssh_username:root 2021-12-01 11:14:58,479 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:14:58,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /data/* 2021-12-01 11:14:58,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:58,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:58,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:58,571 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:14:58,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:58,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:14:59,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:59,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:14:59,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:14:59,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:14:59,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:14:59,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:14:59,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:14:59,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:14:59,224 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.68 - Checking if couchbase-server is installed 2021-12-01 11:14:59,250 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.68 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:59,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:59,319 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.68 - Running systemd command 2021-12-01 11:14:59,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: systemctl start couchbase-server.service 2021-12-01 11:14:59,352 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.68 port:8091 ssh_username:root 2021-12-01 11:14:59,354 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xf60925f2, 0.0.0.0/0.0.0.0:44716] timeout=300.0>) 2021-12-01 11:14:59,358 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:59,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:59,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x6bd72235, 0.0.0.0/0.0.0.0:44568] timeout=300.0>) 2021-12-01 11:15:01,375 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:15:01,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:15:01,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x77b1cf4f, 0.0.0.0/0.0.0.0:40876] timeout=300.0>) 2021-12-01 11:15:03,391 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:15:03,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:15:03,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x5f409ec4, 0.0.0.0/0.0.0.0:50277] timeout=300.0>) 2021-12-01 11:15:05,407 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:15:05,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:15:05,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:15:07,749 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:15:07,766 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:15:07,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> reset_cluster():L562 2021-12-01 11:15:07,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-01 11:15:17,782 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:15:17,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:18,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:15:18,263 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:18,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:15:18,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:15:18,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:15:18,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:15:18,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:15:18,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:15:18,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:15:18,448 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:15:18,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:18,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:15:18,924 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:18,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:15:18,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:15:18,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:15:18,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:15:19,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:15:19,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:15:19,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:15:19,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:15:19,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:19,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:15:19,545 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:19,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:15:19,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:15:19,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:15:19,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:15:19,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:15:19,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:15:19,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:15:19,661 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:15:19,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:20,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:15:20,155 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:20,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:15:20,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:15:20,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:15:20,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:15:20,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:15:20,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:15:20,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:15:20,262 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:15:20,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:20,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:15:20,760 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:20,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:15:20,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:15:20,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:15:20,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:15:20,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:15:20,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:15:20,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:15:20,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /data/* 2021-12-01 11:15:20,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: chown -R couchbase:couchbase /data 2021-12-01 11:15:20,990 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:23,752 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:23,755 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.64_8091 2021-12-01 11:15:23,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /data/* 2021-12-01 11:15:23,756 | infra | INFO | pool-3-thread-1 | [task:start_task:89] Thread 'node_init_task_172.23.110.64_8091' started 2021-12-01 11:15:23,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: chown -R couchbase:couchbase /data 2021-12-01 11:15:23,782 | infra | DEBUG | pool-3-thread-1 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:23,789 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,789 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:23,819 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:23,823 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,823 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:23,865 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,865 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:23,921 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,921 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:24,022 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:24,023 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:24,203 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:24,203 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:24,545 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:24,545 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:25,206 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:25,207 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:26,507 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,509 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:26,684 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:26,684 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.65_8091 2021-12-01 11:15:26,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /data/* 2021-12-01 11:15:26,684 | infra | INFO | pool-3-thread-2 | [task:start_task:89] Thread 'node_init_task_172.23.110.65_8091' started 2021-12-01 11:15:26,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: chown -R couchbase:couchbase /data 2021-12-01 11:15:26,721 | infra | DEBUG | pool-3-thread-2 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:26,730 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,730 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:26,752 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:26,836 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,838 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:26,875 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,875 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:26,931 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,931 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:27,028 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:27,029 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:27,204 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:27,204 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:27,542 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:27,542 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:28,200 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:28,201 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:29,081 | test | DEBUG | pool-3-thread-1 | [task:call:5874] server: ip:172.23.110.64 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266823929856L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.64:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.64', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'80', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:29,082 | test | DEBUG | pool-3-thread-1 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:29,095 | test | DEBUG | pool-3-thread-1 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:29,111 | test | DEBUG | pool-3-thread-1 | [rest_client:init_cluster:847] settings/web params on 172.23.110.64:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:29,496 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,496 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:29,599 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:29,601 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.66_8091 2021-12-01 11:15:29,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /data/* 2021-12-01 11:15:29,602 | infra | INFO | pool-3-thread-3 | [task:start_task:89] Thread 'node_init_task_172.23.110.66_8091' started 2021-12-01 11:15:29,625 | infra | DEBUG | pool-3-thread-3 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:29,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: chown -R couchbase:couchbase /data 2021-12-01 11:15:29,628 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,630 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:29,660 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,660 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:29,664 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:29,698 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,698 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:29,755 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,755 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:29,851 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,851 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:30,029 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:30,029 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:30,365 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:30,365 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:31,023 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:31,023 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:32,069 | test | DEBUG | pool-3-thread-2 | [task:call:5874] server: ip:172.23.110.65 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266667524096L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.65:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.65', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'70', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:32,071 | test | DEBUG | pool-3-thread-2 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:32,084 | test | DEBUG | pool-3-thread-2 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:32,099 | test | DEBUG | pool-3-thread-2 | [rest_client:init_cluster:847] settings/web params on 172.23.110.65:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:32,322 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,323 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:32,486 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:32,487 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.67_8091 2021-12-01 11:15:32,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /data/* 2021-12-01 11:15:32,489 | infra | INFO | pool-3-thread-4 | [task:start_task:89] Thread 'node_init_task_172.23.110.67_8091' started 2021-12-01 11:15:32,512 | infra | DEBUG | pool-3-thread-4 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:32,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: chown -R couchbase:couchbase /data 2021-12-01 11:15:32,515 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,515 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:32,539 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,539 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:32,549 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:32,572 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,572 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:32,625 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,627 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:32,719 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,720 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:32,895 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,895 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:33,232 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:33,232 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:33,885 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:33,885 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:34,897 | test | DEBUG | pool-3-thread-3 | [task:call:5874] server: ip:172.23.110.66 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266885197824L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.66:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.66', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'58', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:34,898 | test | DEBUG | pool-3-thread-3 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:34,911 | test | DEBUG | pool-3-thread-3 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:34,928 | test | DEBUG | pool-3-thread-3 | [rest_client:init_cluster:847] settings/web params on 172.23.110.66:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:35,178 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,180 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:35,461 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:35,463 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.68_8091 2021-12-01 11:15:35,464 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.64_8091 2021-12-01 11:15:35,464 | infra | INFO | pool-3-thread-5 | [task:start_task:89] Thread 'node_init_task_172.23.110.68_8091' started 2021-12-01 11:15:35,464 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.65_8091 2021-12-01 11:15:35,466 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.66_8091 2021-12-01 11:15:35,466 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.67_8091 2021-12-01 11:15:35,492 | infra | DEBUG | pool-3-thread-5 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:35,496 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,497 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:35,525 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,526 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:35,561 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,562 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:35,615 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,615 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:35,707 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,709 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:35,882 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,884 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:36,220 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:36,220 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:36,877 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:36,877 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:37,756 | test | DEBUG | pool-3-thread-4 | [task:call:5874] server: ip:172.23.110.67 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266915176448L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.67:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.67', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'48', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:37,757 | test | DEBUG | pool-3-thread-4 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:37,776 | test | DEBUG | pool-3-thread-4 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:37,792 | test | DEBUG | pool-3-thread-4 | [rest_client:init_cluster:847] settings/web params on 172.23.110.67:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:37,936 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.68_8091 2021-12-01 11:15:38,171 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:38,171 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:40,743 | test | DEBUG | pool-3-thread-5 | [task:call:5874] server: ip:172.23.110.68 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266776530944L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.68:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.68', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'38', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:40,744 | test | DEBUG | pool-3-thread-5 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:40,755 | test | DEBUG | pool-3-thread-5 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:40,770 | test | DEBUG | pool-3-thread-5 | [rest_client:init_cluster:847] settings/web params on 172.23.110.68:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:40,900 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:710] RAM quota was defined less than 100 MB: 2021-12-01 11:15:40,900 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.64: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256114988 kB\n', 'MemAvailable: 259646652 kB\n', 'Buffers: 40 kB\n', 'Cached: 4454364 kB\n', 'SwapCached: 372 kB\n', 'Active: 1546812 kB\n', 'Inactive: 3351568 kB\n', 'Active(anon): 428804 kB\n', 'Inactive(anon): 356008 kB\n', 'Active(file): 1118008 kB\n', 'Inactive(file): 2995560 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4151664 kB\n', 'Dirty: 80 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 443340 kB\n', 'Mapped: 107076 kB\n', 'Shmem: 340788 kB\n', 'Slab: 720556 kB\n', 'SReclaimable: 302152 kB\n', 'SUnreclaim: 418404 kB\n', 'KernelStack: 27760 kB\n', 'PageTables: 11704 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3160164 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 273408 kB\n', 'DirectMap2M: 10870784 kB\n', 'DirectMap1G: 258998272 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.65: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 255937496 kB\n', 'MemAvailable: 259597704 kB\n', 'Buffers: 40 kB\n', 'Cached: 4470784 kB\n', 'SwapCached: 384 kB\n', 'Active: 1529204 kB\n', 'Inactive: 3399516 kB\n', 'Active(anon): 427824 kB\n', 'Inactive(anon): 407444 kB\n', 'Active(file): 1101380 kB\n', 'Inactive(file): 2992072 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4173728 kB\n', 'Dirty: 56 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 458360 kB\n', 'Mapped: 105444 kB\n', 'Shmem: 377368 kB\n', 'Slab: 866172 kB\n', 'SReclaimable: 450812 kB\n', 'SUnreclaim: 415360 kB\n', 'KernelStack: 27504 kB\n', 'PageTables: 12336 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3186320 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 259072 kB\n', 'DirectMap2M: 17176576 kB\n', 'DirectMap1G: 252706816 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.66: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256095312 kB\n', 'MemAvailable: 259560528 kB\n', 'Buffers: 28 kB\n', 'Cached: 4534296 kB\n', 'SwapCached: 580 kB\n', 'Active: 1537848 kB\n', 'Inactive: 3436272 kB\n', 'Active(anon): 438188 kB\n', 'Inactive(anon): 440648 kB\n', 'Active(file): 1099660 kB\n', 'Inactive(file): 2995624 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4092676 kB\n', 'Dirty: 60 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 440312 kB\n', 'Mapped: 101688 kB\n', 'Shmem: 439016 kB\n', 'Slab: 655584 kB\n', 'SReclaimable: 254064 kB\n', 'SUnreclaim: 401520 kB\n', 'KernelStack: 27440 kB\n', 'PageTables: 11672 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3317328 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 0 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 271360 kB\n', 'DirectMap2M: 16115712 kB\n', 'DirectMap1G: 253755392 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.67: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256329400 kB\n', 'MemAvailable: 259836584 kB\n', 'Buffers: 40 kB\n', 'Cached: 4325856 kB\n', 'SwapCached: 132 kB\n', 'Active: 1562548 kB\n', 'Inactive: 3222252 kB\n', 'Active(anon): 430044 kB\n', 'Inactive(anon): 210892 kB\n', 'Active(file): 1132504 kB\n', 'Inactive(file): 3011360 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4179932 kB\n', 'Dirty: 180 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 458972 kB\n', 'Mapped: 118788 kB\n', 'Shmem: 182004 kB\n', 'Slab: 610648 kB\n', 'SReclaimable: 247376 kB\n', 'SUnreclaim: 363272 kB\n', 'KernelStack: 27360 kB\n', 'PageTables: 11572 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 2988336 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 778636 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 8192 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 259072 kB\n', 'DirectMap2M: 21370880 kB\n', 'DirectMap1G: 248512512 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.68: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256200432 kB\n', 'MemAvailable: 259774132 kB\n', 'Buffers: 40 kB\n', 'Cached: 4315464 kB\n', 'SwapCached: 1260 kB\n', 'Active: 1544584 kB\n', 'Inactive: 3211924 kB\n', 'Active(anon): 429680 kB\n', 'Inactive(anon): 215080 kB\n', 'Active(file): 1114904 kB\n', 'Inactive(file): 2996844 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4095276 kB\n', 'Dirty: 836 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 440656 kB\n', 'Mapped: 103072 kB\n', 'Shmem: 203688 kB\n', 'Slab: 769984 kB\n', 'SReclaimable: 346084 kB\n', 'SUnreclaim: 423900 kB\n', 'KernelStack: 27616 kB\n', 'PageTables: 11264 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3075824 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 778636 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 271360 kB\n', 'DirectMap2M: 18212864 kB\n', 'DirectMap1G: 251658240 kB\n'] 2021-12-01 11:15:40,901 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:15:40,901 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:15:40,903 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:15:40,903 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:15:40,904 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:15:40,907 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:15:41,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:41,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:15:41,388 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:41,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:15:41,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:15:41,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:15:41,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:15:41,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:15:41,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:15:41,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:15:41,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:41,562 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.64 port:8091 ssh_username:root: [] 2021-12-01 11:15:41,562 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:15:41,563 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:41,568 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:15:41,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:41,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:15:42,052 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:42,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:15:42,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:15:42,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:15:42,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:15:42,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:15:42,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:15:42,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:15:42,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:42,232 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.65 port:8091 ssh_username:root: [] 2021-12-01 11:15:42,233 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:15:42,233 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:42,236 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:15:42,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:42,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:15:42,697 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:42,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:15:42,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:15:42,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:15:42,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:15:42,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:15:42,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:15:42,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:15:42,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:42,815 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.66 port:8091 ssh_username:root: [] 2021-12-01 11:15:42,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:15:42,815 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:42,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:15:42,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:43,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:15:43,303 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:43,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:15:43,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:15:43,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:15:43,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:15:43,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:15:43,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:15:43,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:15:43,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:43,420 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.67 port:8091 ssh_username:root: [] 2021-12-01 11:15:43,420 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:15:43,421 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:43,424 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:15:43,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:43,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:15:43,930 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:43,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:15:43,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:15:43,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:15:43,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:15:44,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:15:44,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:15:44,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:15:44,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:44,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.68 port:8091 ssh_username:root: [] 2021-12-01 11:15:44,082 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:15:44,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:44,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:361] Wait for all the services to come up after change_env_vars update 2021-12-01 11:15:44,091 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> change_env_variables():L363 2021-12-01 11:15:44,091 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-01 11:15:54,092 | test | INFO | MainThread | [basetestcase:initialize_cluster:439] Cluster C1 initialized 2021-12-01 11:15:54,092 | test | INFO | MainThread | [basetestcase:log_setup_status:658] ========= BaseTestCase setup finished for test #1 SystemTestMagma ========= ############################################################ # # Step 1: Create a 3 node cluster # ############################################################ 2021-12-01 11:15:54,144 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12 2021-12-01 11:15:54,147 | infra | INFO | pool-3-thread-6 | [task:start_task:89] Thread 'Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12' started 2021-12-01 11:15:54,147 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12 2021-12-01 11:15:54,148 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1178] Adding remote node 172.23.110.65:8091 to cluster 172.23.110.64:8091 2021-12-01 11:16:05,678 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1178] Adding remote node 172.23.110.66:8091 to cluster 172.23.110.64:8091 2021-12-01 11:16:09,954 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-01 11:16:10,013 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-01 11:16:10,015 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 0.152801122394 | Cluster node | | 172.23.110.65 | None | | | <--- IN --- | | 172.23.110.66 | None | | | <--- IN --- | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-01 11:16:10,026 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-01 11:16:10,026 | test | INFO | pool-3-thread-6 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-01 11:16:10,042 | infra | DEBUG | pool-3-thread-6 | [task:check:411] Wait before next rebalance progress check 2021-12-01 11:16:10,045 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-01 11:16:10,045 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-01 11:16:20,059 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1649 2021-12-01 11:16:20,061 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-01 11:16:25,071 | test | INFO | pool-3-thread-6 | [task:check:384] Rebalance - status: none, progress: 100 2021-12-01 11:16:25,089 | test | INFO | pool-3-thread-6 | [task:check:443] Rebalance completed with progress: 100% in 15.0759999752 sec 2021-12-01 11:16:25,091 | infra | INFO | pool-3-thread-6 | [task:complete_task:104] Thread 'Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12' completed 2021-12-01 11:16:25,151 | test | INFO | MainThread | [rest_client:set_metadata_purge_interval:2141] Successfully set Metadata Purge Interval to 0.04 ############################################################ # # Step 2: Create required buckets and collections. # ############################################################ 2021-12-01 11:16:25,153 | test | INFO | MainThread | [opd:create_required_buckets:49] Get the available memory quota 2021-12-01 11:16:25,220 | test | INFO | MainThread | [opd:create_required_buckets:57] Create CB buckets 2021-12-01 11:16:25,220 | test | DEBUG | MainThread | [bucket_ready_functions:async_create_bucket:1569] Creating bucket: GleamBookUsers0 2021-12-01 11:16:25,223 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: bucket_GleamBookUsers0_create_task 2021-12-01 11:16:25,223 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for bucket_GleamBookUsers0_create_task 2021-12-01 11:16:25,285 | infra | INFO | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:344] Creating 'membase' bucket GleamBookUsers0 2021-12-01 11:16:25,285 | infra | DEBUG | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:346] http://172.23.110.64:8091/pools/default/buckets with param: storageBackend=magma&fragmentationPercentage=50&replicaIndex=1&maxTTL=0&flushEnabled=1&durabilityMinLevel=none&compressionMode=passive&bucketType=membase&conflictResolutionType=seqno&name=GleamBookUsers0&replicaNumber=1&ramQuotaMB=102400&evictionPolicy=fullEviction 2021-12-01 11:16:25,321 | infra | DEBUG | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:354] 0.04 seconds to create bucket GleamBookUsers0 2021-12-01 11:16:25,322 | infra | INFO | pool-3-thread-7 | [task:complete_task:104] Thread 'bucket_GleamBookUsers0_create_task' completed 2021-12-01 11:16:25,322 | test | DEBUG | MainThread | [bucket_ready_functions:create_bucket:1581] Wait for memcached to accept cbstats or any other bucket request connections 2021-12-01 11:16:25,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> create_bucket():L1583 2021-12-01 11:16:25,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:16:27,400 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 11:16:27,400 | test | INFO | MainThread | [opd:create_required_buckets:79] Rebalance Starts 2021-12-01 11:16:27,430 | test | DEBUG | MainThread | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-01 11:16:27,486 | test | DEBUG | MainThread | [rest_client:rebalance:1436] Rebalance operation started 2021-12-01 11:16:27,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-01 11:16:27,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1567 2021-12-01 11:16:27,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-01 11:16:30,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1649 2021-12-01 11:16:30,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-01 11:16:35,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1567 2021-12-01 11:16:35,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-01 11:16:38,561 | test | INFO | MainThread | [rest_client:monitorRebalance:1576] Rebalance done. Taken 11.0730001926 seconds to complete 2021-12-01 11:16:38,568 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1579 2021-12-01 11:16:38,569 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-12-01 11:16:43,569 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection0 2021-12-01 11:16:43,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:43,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:44,135 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection1 2021-12-01 11:16:44,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:44,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:44,696 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection2 2021-12-01 11:16:44,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:44,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:45,256 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection3 2021-12-01 11:16:45,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:45,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:45,821 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection4 2021-12-01 11:16:45,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:45,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:46,394 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection5 2021-12-01 11:16:46,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:46,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:46,961 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection6 2021-12-01 11:16:47,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:47,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:47,529 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection7 2021-12-01 11:16:47,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:47,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:48,091 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection8 2021-12-01 11:16:48,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:48,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:48,658 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection9 2021-12-01 11:16:48,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:48,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:49,230 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection10 2021-12-01 11:16:49,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:49,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:49,793 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection11 2021-12-01 11:16:49,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:49,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:50,357 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection12 2021-12-01 11:16:50,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:50,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:50,920 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection13 2021-12-01 11:16:50,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:50,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:51,480 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection14 2021-12-01 11:16:51,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:51,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:52,042 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection15 2021-12-01 11:16:52,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:52,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:52,614 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection16 2021-12-01 11:16:52,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:52,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:53,180 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection17 2021-12-01 11:16:53,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:53,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:53,746 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection18 2021-12-01 11:16:53,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:53,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:54,309 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection19 2021-12-01 11:16:54,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:54,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:54,869 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection20 2021-12-01 11:16:54,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:54,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:55,434 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection21 2021-12-01 11:16:55,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:55,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:55,996 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection22 2021-12-01 11:16:56,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:56,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:56,569 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection23 2021-12-01 11:16:56,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:56,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:57,132 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection24 2021-12-01 11:16:57,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:57,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:57,694 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection25 2021-12-01 11:16:57,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:57,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:58,260 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection26 2021-12-01 11:16:58,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:58,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:58,831 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection27 2021-12-01 11:16:58,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:58,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:59,401 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection28 2021-12-01 11:16:59,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:59,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:59,974 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection29 2021-12-01 11:17:00,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:00,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:00,539 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection30 2021-12-01 11:17:00,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:00,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:01,104 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection31 2021-12-01 11:17:01,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:01,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:01,670 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection32 2021-12-01 11:17:01,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:01,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:02,236 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection33 2021-12-01 11:17:02,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:02,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:02,799 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection34 2021-12-01 11:17:02,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:02,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:03,364 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection35 2021-12-01 11:17:03,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:03,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:03,923 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection36 2021-12-01 11:17:03,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:03,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:04,484 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection37 2021-12-01 11:17:04,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:04,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:05,048 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection38 2021-12-01 11:17:05,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:05,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:05,611 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection39 2021-12-01 11:17:05,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:05,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:06,174 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection40 2021-12-01 11:17:06,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:06,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:06,740 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection41 2021-12-01 11:17:06,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:06,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:07,308 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection42 2021-12-01 11:17:07,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:07,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:07,871 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection43 2021-12-01 11:17:07,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:07,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:08,461 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection44 2021-12-01 11:17:08,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:08,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:09,072 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection45 2021-12-01 11:17:09,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:09,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:09,640 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection46 2021-12-01 11:17:09,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:09,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:10,204 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection47 2021-12-01 11:17:10,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:10,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:10,766 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection48 2021-12-01 11:17:10,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:10,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:11,332 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection49 2021-12-01 11:17:11,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:11,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:11,898 | test | DEBUG | MainThread | [opd:create_required_collections:119] Collections list == ['VolumeCollection46', 'VolumeCollection47', 'VolumeCollection44', 'VolumeCollection45', 'VolumeCollection48', 'VolumeCollection49', 'VolumeCollection42', 'VolumeCollection43', 'VolumeCollection40', 'VolumeCollection41', '_default', 'VolumeCollection35', 'VolumeCollection36', 'VolumeCollection33', 'VolumeCollection34', 'VolumeCollection39', 'VolumeCollection37', 'VolumeCollection38', 'VolumeCollection7', 'VolumeCollection6', 'VolumeCollection5', 'VolumeCollection4', 'VolumeCollection3', 'VolumeCollection31', 'VolumeCollection2', 'VolumeCollection32', 'VolumeCollection1', 'VolumeCollection0', 'VolumeCollection30', 'VolumeCollection9', 'VolumeCollection8', 'VolumeCollection19', 'VolumeCollection24', 'VolumeCollection25', 'VolumeCollection22', 'VolumeCollection23', 'VolumeCollection28', 'VolumeCollection29', 'VolumeCollection26', 'VolumeCollection27', 'VolumeCollection20', 'VolumeCollection21', 'VolumeCollection13', 'VolumeCollection14', 'VolumeCollection11', 'VolumeCollection12', 'VolumeCollection17', 'VolumeCollection18', 'VolumeCollection15', 'VolumeCollection16', 'VolumeCollection10'] 2021-12-01 11:17:11,913 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:2037] settings/autoFailover params: failoverServerGroup=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=600&enabled=false&canAbortRebalance=false [ip:172.23.110.67 port:8091 ssh_username:root, ip:172.23.110.68 port:8091 ssh_username:root] 2021-12-01 11:17:11,957 | infra | INFO | MainThread | [BucketOperations_Rest:update_memcached_settings:373] Updating memcached properties 2021-12-01 11:17:11,957 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:374] http://172.23.110.64:8091/pools/default/settings/memcached/global with param: num_reader_threads=disk_io_optimized&num_writer_threads=disk_io_optimized&num_storage_threads=40 2021-12-01 11:17:11,983 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:380] Memcached settings updated ############################################################ # # Step 1: Create 40000000 items sequentially # ############################################################ 2021-12-01 11:17:11,986 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:17:12,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:17:12,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:17:12,476 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:17:12,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:17:12,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:17:12,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:17:12,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:17:12,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:17:12,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:17:12,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:17:12,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-12-01 11:17:12,714 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 ############################################################ # # Step 2: Create 40000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 948.73828125 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 40000000 Final Start: 0 Final End: 40000000 2021-12-01 11:17:13,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:13,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638386234.22 2021-12-01 11:17:14,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:14,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638386235.25 2021-12-01 11:17:15,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:15,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638386236.27 2021-12-01 11:17:16,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:16,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638386237.29 2021-12-01 11:17:17,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:17,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638386238.36 2021-12-01 11:17:18,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:18,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638386239.39 2021-12-01 11:17:19,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:19,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638386240.45 2021-12-01 11:17:20,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:20,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638386241.47 2021-12-01 11:17:21,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:21,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638386242.49 2021-12-01 11:17:22,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:22,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638386243.51 2021-12-01 11:17:23,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:23,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:24,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:24,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:25,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:25,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:26,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:26,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:27,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:27,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:28,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:28,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:29,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:29,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:30,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:30,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:31,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:31,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:32,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:32,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:33,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:33,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:34,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:34,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:35,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:35,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:36,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:36,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:37,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:37,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:38,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:38,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:39,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:39,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:40,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:40,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:42,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:42,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:43,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:43,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:44,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:44,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:45,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:45,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:46,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:46,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:47,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:47,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:48,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:48,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:49,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:49,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:50,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:50,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:51,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:51,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:52,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:52,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:53,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:53,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:54,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:54,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:55,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:55,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:56,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:56,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:57,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:57,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:58,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:58,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:59,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:59,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:00,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:00,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:01,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:01,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:02,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:02,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:03,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:03,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection44_8_1638386236.27 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638386244.52 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638386241.47 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638386245.54 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638386237.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638386246.55 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638386234.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638386247.57 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638386235.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638386248.58 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638386243.51 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638386249.62 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638386238.36 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638386250.67 Loader_GleamBookUsers0__default_VolumeCollection49_5_1638386239.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638386251.69 Loader_GleamBookUsers0__default_VolumeCollection40_2_1638386242.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638386252.7 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638386240.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638386253.71 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638386245.54 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638386254.73 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638386246.55 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638386255.75 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638386244.52 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638386256.76 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638386247.57 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638386257.78 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638386249.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638386258.8 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638386251.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638386259.81 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638386253.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638386260.83 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638386250.67 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638386261.92 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638386248.58 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638386263.24 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638386252.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638386264.26 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638386254.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638386265.27 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638386255.75 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638386266.29 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638386258.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638386267.3 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638386257.78 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638386268.31 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638386260.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638386269.33 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638386256.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638386270.34 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638386263.24 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638386271.35 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638386259.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638386272.37 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638386261.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638386273.38 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638386264.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638386274.39 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638386265.27 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638386275.41 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638386267.3 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638386276.42 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638386269.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638386277.44 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638386271.35 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638386278.45 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638386266.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638386279.47 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638386273.38 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638386280.48 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638386268.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638386281.5 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638386270.34 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638386282.51 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638386272.37 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638386283.52 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638386275.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638386284.54 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638386274.39 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638386279.47 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638386277.44 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638386281.5 is completed! Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638386276.42 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638386278.45 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638386283.52 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638386280.48 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638386282.51 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638386284.54 is completed! 2021-12-01 15:19:49,391 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 15:19:49,397 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:19:49,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:49,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:19:49,855 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:49,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:19:49,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:19:49,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:19:49,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:19:49,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:19:49,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:19:49,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:19:50,010 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 15:19:50,012 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01 2021-12-01 15:19:50,013 | infra | INFO | pool-3-thread-8 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01' started 2021-12-01 15:19:50,015 | infra | INFO | pool-3-thread-8 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01' completed 2021-12-01 15:19:50,017 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:19:50,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:50,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:19:50,480 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:50,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:19:50,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:19:50,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:19:50,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:19:50,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:19:50,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:19:50,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:19:50,631 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 15:19:50,632 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63 2021-12-01 15:19:50,634 | infra | INFO | pool-3-thread-9 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63' started 2021-12-01 15:19:50,635 | infra | INFO | pool-3-thread-9 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63' completed 2021-12-01 15:19:50,638 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:19:50,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:51,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:19:51,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:51,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:19:51,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:19:51,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:19:51,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:19:51,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:19:51,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:19:51,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:19:51,243 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 15:19:51,243 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24 2021-12-01 15:19:51,244 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01 2021-12-01 15:19:51,244 | infra | INFO | pool-3-thread-10 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24' started 2021-12-01 15:19:51,246 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:19:51,246 | infra | INFO | pool-3-thread-10 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24' completed 2021-12-01 15:19:51,246 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63 2021-12-01 15:19:51,247 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:19:51,247 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24 2021-12-01 15:19:51,249 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 15:19:51,289 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 20018.2, u'diskFetches': 0, u'dataUsed': 4506574479816L, u'itemCount': 1999850916, u'quotaPercentUsed': 79.13226621349652, u'storageTotals': {u'hdd': {u'usedByData': 4555016243935L, u'used': 4866615740985L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 31109194249671L}, u'ram': {u'usedByData': 254902871584L, u'used': 332909170688L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 4555016243935L, u'vbActiveNumNonResident': 1860183007, u'memUsed': 254902871584L} 2021-12-01 15:19:51,651 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 1999850916 | 300.00 GiB | 237.40 GiB | 4.14 TiB | 6.98918700806 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-01 15:19:51,653 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 0 | 2000000000 | 0-0 | 0-40000000 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2021-12-01 15:19:52,023 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 6.98918700806 2021-12-01 15:19:52,026 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:19:52,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:52,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:19:52,499 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:52,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:19:52,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:19:52,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:19:52,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:19:52,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:19:52,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:19:52,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:19:52,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 15:19:54,799 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:19:54,802 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:19:54,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:55,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:19:55,296 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:55,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:19:55,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:19:55,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:19:55,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:19:55,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:19:55,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:19:55,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:19:55,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 15:19:55,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 15:19:55,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 15:19:55,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 15:19:56,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 15:19:57,341 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:19:57,345 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:19:57,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:57,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:19:57,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:57,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:19:57,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:19:57,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:19:57,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:19:57,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:19:57,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:19:57,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:19:57,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 15:19:59,947 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 15:19:59,948 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 4430238MB 2021-12-01 15:19:59,948 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 6974MB 2021-12-01 15:19:59,948 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 206798MB 2021-12-01 15:19:59,950 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 4223053MB 2021-12-01 15:19:59,954 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:20:00,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:20:00,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:20:00,424 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:20:00,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:20:00,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:20:00,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:20:00,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:20:00,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:20:00,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:20:00,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:20:00,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 15:20:00,655 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-01 15:20:00,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 15:20:01,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 15:20:01,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 15:20:02,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 15:20:02,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 15:20:02,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 15:20:03,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 15:20:03,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 15:20:04,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 15:20:04,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 15:20:05,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 15:20:05,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 15:20:05,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 15:20:06,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 15:20:06,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 15:20:07,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 15:20:07,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 15:20:08,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 15:20:08,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 15:20:09,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 15:20:09,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 15:20:10,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 15:20:10,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 15:20:10,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 15:20:11,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 15:20:11,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 15:20:11,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 15:20:12,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 15:20:12,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 15:20:13,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 15:20:13,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 15:20:14,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 15:20:14,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 15:20:14,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 15:20:15,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 15:20:15,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 15:20:16,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 15:20:16,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 15:20:17,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 15:20:17,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 15:20:18,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 15:20:18,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 15:20:19,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 15:20:19,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 15:20:20,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 15:20:20,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 15:20:21,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 15:20:21,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 15:20:21,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 15:20:22,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 15:20:22,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 15:20:23,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 15:20:23,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 15:20:23,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 15:20:24,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 15:20:24,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 15:20:25,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 15:20:25,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 15:20:26,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 15:20:26,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 15:20:27,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 15:20:27,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 15:20:27,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 15:20:28,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 15:20:28,816 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:20:28,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:20:28,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:20:29,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:20:29,299 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:20:29,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:20:29,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:20:29,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:20:29,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:20:29,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:20:29,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:20:29,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:20:29,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 15:20:29,476 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-01 15:20:29,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 15:20:29,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 15:20:30,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 15:20:30,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 15:20:31,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 15:20:31,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 15:20:31,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 15:20:32,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 15:20:32,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 15:20:32,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 15:20:33,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 15:20:33,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 15:20:34,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 15:20:34,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 15:20:35,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 15:20:35,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 15:20:36,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 15:20:36,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 15:20:36,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 15:20:37,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 15:20:37,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 15:20:38,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 15:20:38,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 15:20:39,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 15:20:39,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 15:20:39,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 15:20:40,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 15:20:40,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 15:20:41,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 15:20:41,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 15:20:42,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 15:20:42,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 15:20:42,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 15:20:43,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 15:20:43,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 15:20:44,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 15:20:44,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 15:20:45,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 15:20:45,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 15:20:46,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 15:20:46,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 15:20:47,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 15:20:47,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 15:20:47,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 15:20:48,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 15:20:48,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 15:20:48,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 15:20:49,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 15:20:49,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 15:20:50,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 15:20:50,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 15:20:50,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 15:20:51,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 15:20:51,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 15:20:51,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 15:20:52,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 15:20:52,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 15:20:53,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 15:20:53,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 15:20:54,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 15:20:54,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 15:20:55,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 15:20:55,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 15:20:55,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 15:20:56,354 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 15:20:56,357 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:20:56,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:20:56,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:20:56,809 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:20:56,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:20:56,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:20:56,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:20:56,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:20:56,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:20:56,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:20:56,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:20:57,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 15:20:57,081 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-01 15:20:57,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 15:20:57,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 15:20:58,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 15:20:58,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 15:20:58,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 15:20:59,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 15:20:59,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 15:21:00,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 15:21:00,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 15:21:01,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 15:21:01,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 15:21:02,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 15:21:02,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 15:21:03,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 15:21:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 15:21:04,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 15:21:04,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 15:21:05,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 15:21:05,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 15:21:06,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 15:21:06,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 15:21:07,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 15:21:07,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 15:21:08,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 15:21:08,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 15:21:09,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 15:21:09,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 15:21:10,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 15:21:10,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 15:21:11,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 15:21:11,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 15:21:12,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 15:21:12,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 15:21:13,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 15:21:13,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 15:21:13,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 15:21:14,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 15:21:14,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 15:21:15,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 15:21:15,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 15:21:16,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 15:21:16,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 15:21:17,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 15:21:17,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 15:21:18,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 15:21:18,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 15:21:19,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 15:21:19,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 15:21:20,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 15:21:20,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 15:21:21,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 15:21:21,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 15:21:22,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 15:21:22,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 15:21:23,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 15:21:23,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 15:21:23,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 15:21:24,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 15:21:24,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 15:21:25,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 15:21:25,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 15:21:26,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 15:21:26,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 15:21:27,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 15:21:27,831 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:21:27,832 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 21213.19921875, u'ReplayWALOffset': u'2913:6975488', u'NTablesCreated': 153288, u'RecentReadIOAmpSet': 0.018630648031830788, u'ReadIOAmp': 0.29530394077301025, u'BlockCacheMisses': 5314521, u'NFlushes': 51179, u'RecentWriteAmp': 3.725177526473999, u'NumSyncFlushes': 0, u'TotalMemUsed': 89886137, u'NBloomFilterHits': 287060085, u'Fragmentation': 0.0022416117135435343, u'NumNormalFlushes': 26312, u'RecentBlockCacheHitRatio': 72.69005584716797, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 47916, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 369688, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 183135030, u'CheckpointOverhead': 0, u'NItemsGet': 0, u'NWriteIOs': 102151, u'NReadBytes': 0, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 229616612, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 11732074496L, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 45923322952L, u'LogicalSize': 23464148992L, u'SizeRatio': 0.5109418805891989, u'NWriteBytes': 23464148992L, u'BloomFilterMemUsed': 0, u'NumTables': 552, u'WastedSpace': 502487516, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 23464148992L}], u'NItemsGarbageCollected': 0, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 141046715, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 58272, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 0, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 171174, u'CompactWriter': {u'NDataBlockWrites': 0, u'NWriteBytes': 0, u'NIndexBlockWrites': 0, u'NBytesIncoming': 0, u'NSyncs': 0, u'NInserts': 0, u'NWriteIOs': 0}, u'NWriteBytes': 23502967725L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 23464148992L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.7587484089319728, u'BufferMemUsed': 502756, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 23464148992L, u'NTableFiles': 552, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 150536, u'TreeSnapshotMemoryUsed': 100512, u'TotalDiskUsage': 30525743104L, u'BytesIncoming': 30975969700L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 0, u'NItemsPut': 21426350, u'NSyncs': 24867, u'CompactReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'ActiveDiskUsage': 23464148992L, u'NTablesDeleted': 0, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 7157756, u'NWriteBytes': 23502967725L, u'NIndexBlockWrites': 162867, u'NBytesIncoming': 31168562446L, u'NSyncs': 24867, u'NInserts': 21426182, u'NWriteIOs': 102151}}, u'NReadBytes': 21385639100L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 191.23333740234375, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 153066588, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5062, u'NTxns': 3114851, u'NWriteBytes': 39639216128L, u'NSegments': 2913, u'NSyncs': 3114851, u'NLoggedBytes': 39639216128L, u'NActiveSegments': 1, u'NWrites': 3114897, u'NInserts': 47395296, u'DiskUsed': 7516160}, u'NGets': 9742505, u'NReadBytesSet': 1508780968, u'NReadBytesCompact': 11584179160L, u'IndexResidentRatio': 0.07353949546813965, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 77.49811553955078, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 9801459940L, u'BloomFilterMemUsedRatio': 0.12916189432144165, u'NWriteBytes': 76684412582L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.07872121036052704, u'WriteAmp': 2.3573191165924072, u'NBlocksCached': 23179, u'LogicalDataSize': 24201450198L, u'ActiveBloomFilterMemUsed': 19255656, u'NFileCountCompacts': 5666, u'NTableFiles': 1433, u'FlushQueueSize': 72, u'TableObjectMemUsed': 369024, u'BlockCacheQuota': 103093474, u'TotalDiskUsage': 31424991232L, u'NPurgerRuns': 44, u'ReadIOAmpSet': 0.0, u'NSyncs': 74601, u'NReadIOsSetPerSec': 4.0, u'NReadIOsPerSec': 191.8000030517578, u'NTablesDeleted': 104707, u'BloomFilterFPR': 0.0183258093893528, u'NBloomFilterHitsPerSec': 2508.566650390625, u'WriteCacheMemUsed': 748616, u'RecentBloomFilterFPR': 0.02272520586848259, u'BytesPerRead': 851.185546875, u'BytesIncomingPerSec': 406934.78125, u'NReadIOsGetPerSec': 7.633333206176758, u'NGetStats': 224902, u'TotalIndexBlocksSize': 195376948, u'NGetStatsPerSec': 10.433333396911621, u'NWriteBytesCompact': 11364696064L, u'NReadIOsSet': 525330, u'ActiveDataSize': 24201450198L, u'NWriteBytesCompactPerSec': 482781.875, u'NBloomFilterMissesPerSec': 58.33333206176758, u'NCommitBatches': 3114851, u'NGetStatsComputed': 224902, u'NWriteIOs': 3391685, u'NReadIOsGet': 3402330, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 25773368, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 214.6999969482422, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 142848, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 67679188, u'NWriterCompacts': 0, u'NReadIOs': 7757896, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 270012357, u'NTablesCreated': 54643, u'ReadIOAmp': 0.1089764445950934, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 287060085, u'TotalIndexBlocksSize': 12231490, u'CheckpointOverhead': 16527360, u'NItemsGet': 31168855, u'NWriteIOs': 99041, u'NReadBytes': 21328688424L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2518820, u'SizeRatio': 0.05459395321932706, u'NWriteBytes': 1740005376, u'BloomFilterMemUsed': 34736, u'NumTables': 95, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2387968}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 29892154, u'SizeRatio': 0.6478949893604625, u'NWriteBytes': 3485671424L, u'BloomFilterMemUsed': 920528, u'NumTables': 78, u'WastedSpace': 36600, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 29847552}, {u'TargetSize': 79154773, u'LogicalSize': 54372138, u'SizeRatio': 0.6869091520229614, u'NWriteBytes': 4839501824L, u'BloomFilterMemUsed': 1664264, u'NumTables': 55, u'WastedSpace': 134314, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 54476800}, {u'TargetSize': 791547796, u'LogicalSize': 791547796, u'SizeRatio': 1.0, u'NWriteBytes': 3137851392L, u'BloomFilterMemUsed': 16634664, u'NumTables': 331, u'WastedSpace': 2306104, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 793841664}], u'NItemsGarbageCollected': 0, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 12014653, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 64128, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 7650991, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 203017, u'CompactWriter': {u'NDataBlockWrites': 4117976, u'NWriteBytes': 11341651968L, u'NIndexBlockWrites': 55324, u'NBytesIncoming': 15660663456L, u'NSyncs': 0, u'NInserts': 270012357, u'NWriteIOs': 26258}, u'NWriteBytes': 13114081905L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 3396671, u'NReadBytes': 9793341767L, u'NDataBlockReads': 22054008, u'NBytesOutgoing': 0, u'NIndexBlockReads': 31724585}, u'NBloomFilterMisses': 5358813, u'NItemsMerged': 0, u'LogicalDiskSize': 791547796, u'TotalBloomFilterMemUsed': 19260352, u'WriteAmp': 12.490929421641386, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 19254192, u'LogicalDataSize': 737175658, u'NTableFiles': 668, u'NFileCountCompacts': 2853, u'TableObjectMemUsed': 165664, u'TreeSnapshotMemoryUsed': 103392, u'TotalDiskUsage': 897503232, u'BytesIncoming': 1049888400, u'NItemsDel': 0, u'PendingCheckpointBytes': 622592, u'NCompacts': 3402, u'NItemsPut': 21426350, u'NSyncs': 24867, u'CompactReader': {u'NReadIOs': 4254320, u'NReadBytes': 11535346657L, u'NDataBlockReads': 4155436, u'NBytesOutgoing': 0, u'NIndexBlockReads': 98884}, u'ActiveDiskUsage': 880553984, u'NTablesDeleted': 54084, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.018325809435202782, u'Writer': {u'NDataBlockWrites': 362908, u'NWriteBytes': 1772429937, u'NIndexBlockWrites': 47916, u'NBytesIncoming': 1242715806, u'NSyncs': 24867, u'NInserts': 21426182, u'NWriteIOs': 72783}}, u'NReadBytesCompactPerSec': 479131.65625, u'TableMetaMemUsed': 435313, u'NDeletes': 0, u'BlockCacheHitsPerSec': 509.0, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 5358813, u'LogicalDiskSize': 24255822336L, u'NSets': 21426350, u'TotalBloomFilterMemUsed': 19262016, u'BufferMemUsed': 502756, u'RecentReadAmp': 0.0, u'NGetsPerSec': 96.96666717529297, u'NInserts': 0, u'NCommitBatchesPerSec': 186.10000610351562, u'FragmentationWithOverhead': 0.2298661321401596, u'NCompactsPerSec': 0.20000000298023224, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 500344.875, u'TreeSnapshotMemoryUsed': 249184, u'BytesIncoming': 32530347073L, u'NCompacts': 6215, u'NumPersistentFlushes': 24867, u'NWriteIOsPerSec': 190.93333435058594, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 218.76795959472656, u'ActiveDiskUsage': 24345571328L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 10.433333396911621, u'NReadBytesSetPerSec': 11040.7001953125, u'BlockCacheHits': 18303594, u'NWriteBytesPerSec': 1515904.25, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 50729, u'ReadIOAmp': 0.0009073231673803613, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 10428, u'CheckpointOverhead': 729088, u'NItemsGet': 6237028, u'NWriteIOs': 75596, u'NReadBytes': 56950676, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 860988, u'SizeRatio': 0.018661412325772373, u'NWriteBytes': 392572928, u'BloomFilterMemUsed': 760, u'NumTables': 95, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 778240}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125548, u'SizeRatio': 0.0027211796153675427, u'NWriteBytes': 23044096, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 16623381, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 5220, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 20448, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 106905, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 61122, u'CompactWriter': {u'NDataBlockWrites': 2813, u'NWriteBytes': 23044096, u'NIndexBlockWrites': 2813, u'NBytesIncoming': 12441481, u'NSyncs': 0, u'NInserts': 149089, u'NWriteIOs': 2813}, u'NWriteBytes': 428146824, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5659, u'NReadBytes': 8118173, u'NDataBlockReads': 3103175, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3103175}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125548, u'TotalBloomFilterMemUsed': 1664, u'WriteAmp': 0.23635017555759222, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1464, u'LogicalDataSize': 125548, u'NTableFiles': 213, u'NFileCountCompacts': 2813, u'TableObjectMemUsed': 52824, u'TreeSnapshotMemoryUsed': 45280, u'TotalDiskUsage': 1744896, u'BytesIncoming': 1811493573, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 2813, u'NItemsPut': 16624407, u'NSyncs': 24867, u'CompactReader': {u'NReadIOs': 101246, u'NReadBytes': 48832503, u'NDataBlockReads': 50623, u'NBytesOutgoing': 0, u'NIndexBlockReads': 50623}, u'ActiveDiskUsage': 868352, u'NTablesDeleted': 50623, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 47916, u'NWriteBytes': 405102728, u'NIndexBlockWrites': 47916, u'NBytesIncoming': 34639020, u'NSyncs': 24867, u'NInserts': 438355, u'NWriteIOs': 72783}}}}} 2021-12-01 15:21:27,841 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.0022416117135435343, 0.0022941313218325377, 0.002015193458646536, 0.002324556466192007, 0.0021804943680763245, 0.002765642013400793, 0.0024411696940660477, 0.0021549160592257977, 0.0023552137427031994, 0.0020246657077223063, 0.0023817908950150013, 0.002493878360837698, 0.0025022635236382484, 0.002096407348290086, 0.002557255793362856, 0.0025922060012817383, 0.0024967764038592577, 0.0023721959441900253, 0.0024138230364769697, 0.0027135666459798813, 0.002201174385845661, 0.0026931038592010736, 0.002556746592745185, 0.002837071195244789, 0.0020925782155245543, 0.002808792982250452, 0.0026731863617897034, 0.0022596511989831924, 0.002629559952765703, 0.0022674344945698977, 0.0026121793780475855, 0.0021393971983343363, 0.0026932943146675825, 0.0023969365283846855, 0.002872421173378825, 0.0029118629172444344, 0.0025225942954421043, 0.001633833278901875, 0.0022486140951514244, 0.0025926704984158278, 0.002632267540320754, 0.0025036316365003586, 0.0024240112397819757, 0.0018561369506642222, 0.0021922013256698847, 0.0022548241540789604, 0.0023967749439179897, 0.002344548935070634, 0.002319935243576765, 0.0019873641431331635, 0.002352714305743575, 0.0029873005114495754, 0.0021521528251469135, 0.0022172534372657537, 0.0023673675023019314, 0.0017370587447658181, 0.0019436674192547798, 0.0022926563397049904, 0.0023635465186089277, 0.0017820706125348806, 0.002538592554628849, 0.0018720794469118118, 0.002089310670271516, 0.002391811925917864], '172.23.110.66': [0.0024939149152487516, 0.0021007375326007605, 0.002611503703519702, 0.0021880005951970816, 0.002742948243394494, 0.002491206396371126, 0.002252149861305952, 0.0022760566789656878, 0.002282000845298171, 0.0021774868946522474, 0.0027259541675448418, 0.002180114621296525, 0.0026868656277656555, 0.0024218971375375986, 0.0022028167732059956, 0.002307652961462736, 0.00247765495441854, 0.002475747372955084, 0.002730895997956395, 0.0026096301153302193, 0.0027981132734566927, 0.0025442405603826046, 0.0026306987274438143, 0.002329097827896476, 0.00251530553214252, 0.0023596358951181173, 0.0023948841262608767, 0.002318717073649168, 0.0021518648136407137, 0.0024667561519891024, 0.0016075385501608253, 0.0022604777477681637, 0.0024134665727615356, 0.0028411527164280415, 0.0022247275337576866, 0.0024975701235234737, 0.0025673522613942623, 0.0018893826054409146, 0.0024243383668363094, 0.0022212876938283443, 0.0023614924866706133, 0.002471758285537362, 0.002455576555803418, 0.0028461795300245285, 0.002510031685233116, 0.0020611744839698076, 0.002206519478932023, 0.00224901526235044, 0.002127568470314145, 0.0024237323086708784, 0.0023756232112646103, 0.002508356934413314, 0.002504774136468768, 0.002007761737331748, 0.0025947229005396366, 0.00267780595459044, 0.00244826078414917, 0.00258748815394938, 0.002612530020996928, 0.0025073832366615534, 0.002400142140686512, 0.002614497672766447, 0.0021753450855612755, 0.0020475347992032766], '172.23.110.64': [0.0022234146017581224, 0.002527899108827114, 0.0020567569881677628, 0.0024150044191628695, 0.0025121888611465693, 0.0025077685713768005, 0.002370118396356702, 0.001972913509234786, 0.00241631967946887, 0.0024957263376563787, 0.0022888134699314833, 0.0023345232475548983, 0.0020164253655821085, 0.002202080562710762, 0.00229130988009274, 0.001682913163676858, 0.0025677455123513937, 0.0025699862744659185, 0.0018582910997793078, 0.002481174888089299, 0.0016975300386548042, 0.00261866906657815, 0.0024680288042873144, 0.0025963953230530024, 0.0020359940826892853, 0.002601386746391654, 0.0021907964255660772, 0.0021182314958423376, 0.002775561762973666, 0.002599793951958418, 0.002161510055884719, 0.0027297334745526314, 0.0027865346055477858, 0.0018869200721383095, 0.002287589479237795, 0.002590629504993558, 0.0022109798155725002, 0.0017330119153484702, 0.002267823787406087, 0.002528937766328454, 0.0020939791575074196, 0.0026525924913585186, 0.0028460004832595587, 0.0026993220672011375, 0.002425026148557663, 0.0020654022227972746, 0.0027526472695171833, 0.0017714494606480002, 0.0023511487524956465, 0.0024831565096974373, 0.0024050248321145773, 0.0024220224004238844, 0.0025446289218962193, 0.0027132942341268063, 0.0027059807907789946, 0.0018588807433843613, 0.002387530170381069, 0.001955490792170167, 0.002727484330534935, 0.0023606563918292522, 0.0024163336493074894, 0.0020715356804430485, 0.0025543973315507174, 0.0020314499270170927]} 2021-12-01 15:21:28,180 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 1.0 2021-12-01 15:21:28,522 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 1.0 2021-12-01 15:21:28,868 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 1.0 2021-12-01 15:21:28,868 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 1.0, '172.23.110.66': 1.0, '172.23.110.64': 1.0} 2021-12-01 15:21:28,871 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:21:28,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:21:29,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:21:29,357 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:21:29,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:21:29,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:21:29,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:21:29,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:21:29,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:21:29,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:21:29,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:21:29,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 15:21:29,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-01 15:21:29,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 15:21:29,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:29,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 15:21:29,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:29,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:29,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 15:21:29,803 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-01 15:21:29,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 15:21:29,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:29,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 15:21:29,848 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-01 15:21:29,848 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:21:29,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:21:29,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:21:30,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:21:30,322 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:21:30,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:21:30,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:21:30,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:21:30,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:21:30,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:21:30,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:21:30,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:21:30,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 15:21:30,476 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-01 15:21:30,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 15:21:30,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:30,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 15:21:30,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:30,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:30,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 15:21:30,710 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-01 15:21:30,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 15:21:30,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:30,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 15:21:30,762 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-01 15:21:30,763 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:21:30,766 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:21:30,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:21:31,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:21:31,273 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:21:31,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:21:31,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:21:31,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:21:31,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:21:31,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:21:31,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:21:31,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:21:31,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 15:21:31,430 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-01 15:21:31,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 15:21:31,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:31,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 15:21:31,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:31,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:31,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 15:21:31,661 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-01 15:21:31,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 15:21:31,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:31,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 15:21:31,726 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-01 15:21:31,727 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 2.1: Update 40000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 6882.12890625 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 40000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 0 Final End: 40000000 2021-12-01 15:21:31,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:31,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638400892.8 2021-12-01 15:21:32,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:32,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638400893.81 2021-12-01 15:21:33,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:33,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638400894.82 2021-12-01 15:21:34,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:34,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638400895.84 2021-12-01 15:21:35,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:35,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638400896.85 2021-12-01 15:21:36,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:36,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638400897.92 2021-12-01 15:21:37,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:37,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638400898.94 2021-12-01 15:21:38,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:38,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638400899.95 2021-12-01 15:21:39,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:39,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638400900.96 2021-12-01 15:21:40,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:40,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638400901.98 2021-12-01 15:21:41,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:41,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:43,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:43,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:44,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:44,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:45,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:45,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:46,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:46,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:47,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:47,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:48,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:48,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:49,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:49,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:50,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:50,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:51,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:51,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:52,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:52,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:53,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:53,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:54,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:54,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:55,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:55,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:56,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:56,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:57,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:57,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:58,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:58,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:59,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:59,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:00,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:00,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:01,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:01,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:02,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:02,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:03,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:03,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:04,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:04,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:05,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:05,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:06,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:06,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:07,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:07,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:08,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:08,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:09,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:09,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:10,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:10,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:11,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:11,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:12,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:12,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:13,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:13,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:14,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:14,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:15,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:15,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:16,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:16,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:17,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:17,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:18,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:18,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:19,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:19,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:20,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:20,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:21,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:21,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection49_5_1638400897.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638400902.99 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638400893.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638400904.01 Loader_GleamBookUsers0__default_VolumeCollection44_8_1638400894.82 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638400905.02 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638400898.94 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638400906.06 Loader_GleamBookUsers0__default_VolumeCollection40_2_1638400900.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638400907.08 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638400901.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638400908.09 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638400892.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638400909.11 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638400895.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638400910.13 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638400896.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638400911.14 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638400899.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638400912.2 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638400906.06 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638400913.21 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638400904.01 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638400914.23 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638400908.09 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638400915.24 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638400912.2 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638400916.26 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638400905.02 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638400917.27 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638400902.99 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638400918.29 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638400907.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638400919.37 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638400909.11 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638400920.39 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638400911.14 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638400921.4 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638400910.13 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638400922.41 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638400914.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638400923.43 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638400916.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638400924.44 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638400918.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638400925.45 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638400920.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638400926.47 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638400913.21 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638400927.49 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638400915.24 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638400928.5 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638400917.27 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638400929.52 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638400919.37 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638400930.53 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638400921.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638400931.55 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638400924.44 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638400932.56 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638400922.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638400933.57 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638400926.47 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638400934.59 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638400923.43 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638400935.6 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638400928.5 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638400936.61 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638400925.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638400937.62 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638400930.53 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638400938.64 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638400927.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638400939.65 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638400932.56 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638400940.66 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638400929.52 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638400941.68 Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638400934.59 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638400942.69 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638400933.57 is completed! Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638400931.55 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638400936.61 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638400935.6 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638400938.64 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638400940.66 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638400937.62 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638400939.65 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638400942.69 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638400941.68 is completed! 2021-12-01 19:20:58,312 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 19:20:58,316 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:20:58,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:20:58,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:20:58,802 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:20:58,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:20:58,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:20:58,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:20:58,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:20:58,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:20:58,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:20:58,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:20:58,971 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 19:20:58,973 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97 2021-12-01 19:20:58,973 | infra | INFO | pool-3-thread-11 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97' started 2021-12-01 19:20:58,976 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 19:20:58,979 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:20:59,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:20:59,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:20:59,443 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:20:59,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:20:59,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:20:59,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:20:59,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:20:59,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:20:59,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:20:59,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:20:59,562 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 19:20:59,566 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56 2021-12-01 19:20:59,568 | infra | INFO | pool-3-thread-12 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56' started 2021-12-01 19:20:59,569 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 19:20:59,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:20:59,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:20:59,766 | test | DEBUG | pool-3-thread-11 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 19:20:59,766 | test | DEBUG | pool-3-thread-11 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-01 19:20:59,766 | infra | INFO | pool-3-thread-11 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97' completed 2021-12-01 19:20:59,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:21:00,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:00,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:21:00,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:21:00,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:21:00,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:21:00,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:21:00,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:21:00,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:21:00,206 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 19:21:00,207 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21 2021-12-01 19:21:00,207 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97 2021-12-01 19:21:00,207 | infra | INFO | pool-3-thread-13 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21' started 2021-12-01 19:21:00,210 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:21:00,210 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 19:21:00,210 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56 2021-12-01 19:21:00,354 | test | DEBUG | pool-3-thread-12 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 19:21:00,355 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-01 19:21:00,355 | infra | INFO | pool-3-thread-12 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56' completed 2021-12-01 19:21:00,355 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:21:00,355 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21 2021-12-01 19:21:01,042 | test | DEBUG | pool-3-thread-13 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 19:21:01,042 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-01 19:21:01,042 | infra | INFO | pool-3-thread-13 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21' completed 2021-12-01 19:21:01,043 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 19:21:01,082 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 21826.3, u'diskFetches': 0, u'dataUsed': 3768630891642L, u'itemCount': 2000000000, u'quotaPercentUsed': 82.16366338481505, u'storageTotals': {u'hdd': {u'usedByData': 6785870952984L, u'used': 6940165511574L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 29035644479082L}, u'ram': {u'usedByData': 264667685368L, u'used': 354878623744L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 6785870952984L, u'vbActiveNumNonResident': 1862849343, u'memUsed': 264667685368L} 2021-12-01 19:21:01,433 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 2000000000 | 300.00 GiB | 246.49 GiB | 6.17 TiB | 6.86019555 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ 2021-12-01 19:21:01,434 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 2000000000 | 2000000000 | 0-40000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2021-12-01 19:21:01,798 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 6.86019555 2021-12-01 19:21:01,801 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:21:01,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:02,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:21:02,305 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:02,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:21:02,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:21:02,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:21:02,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:21:02,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:21:02,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:21:02,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:21:02,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 19:21:05,259 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:21:05,263 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:21:05,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:05,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:21:05,737 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:05,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:21:05,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:21:05,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:21:05,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:21:05,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:21:05,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:21:05,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:21:05,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 19:21:08,480 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:21:08,483 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:21:08,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:08,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:21:08,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:08,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:21:08,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:21:09,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:21:09,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:21:09,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:21:09,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:21:09,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:21:09,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 19:21:11,618 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 19:21:11,619 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 6485429MB 2021-12-01 19:21:11,621 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 8650MB 2021-12-01 19:21:11,621 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 192015MB 2021-12-01 19:21:11,621 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 6293018MB 2021-12-01 19:21:11,624 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:21:11,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:12,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:21:12,132 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:12,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:21:12,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:21:12,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:21:12,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:21:12,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:21:12,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:21:12,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:21:12,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 19:21:12,357 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-01 19:21:12,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 19:21:12,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 19:21:13,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 19:21:13,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 19:21:14,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 19:21:14,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 19:21:14,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 19:21:15,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 19:21:15,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 19:21:16,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 19:21:16,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 19:21:17,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 19:21:17,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 19:21:17,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 19:21:18,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 19:21:18,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 19:21:18,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 19:21:19,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 19:21:19,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 19:21:20,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 19:21:20,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 19:21:21,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 19:21:21,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 19:21:22,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 19:21:22,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 19:21:23,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 19:21:23,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 19:21:24,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 19:21:24,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 19:21:25,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 19:21:25,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 19:21:26,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 19:21:26,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 19:21:27,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 19:21:27,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 19:21:27,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 19:21:28,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 19:21:28,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 19:21:29,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 19:21:29,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 19:21:30,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 19:21:30,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 19:21:30,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 19:21:31,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 19:21:31,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 19:21:32,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 19:21:32,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 19:21:33,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 19:21:33,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 19:21:33,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 19:21:34,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 19:21:34,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 19:21:34,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 19:21:35,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 19:21:35,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 19:21:36,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 19:21:36,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 19:21:37,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 19:21:37,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 19:21:37,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 19:21:38,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 19:21:38,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 19:21:39,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 19:21:39,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 19:21:39,785 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:21:39,789 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:21:39,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:40,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:21:40,265 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:40,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:21:40,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:21:40,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:21:40,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:21:40,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:21:40,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:21:40,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:21:40,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 19:21:40,464 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-01 19:21:40,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 19:21:40,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 19:21:41,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 19:21:41,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 19:21:42,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 19:21:42,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 19:21:43,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 19:21:43,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 19:21:44,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 19:21:44,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 19:21:45,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 19:21:45,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 19:21:46,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 19:21:46,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 19:21:46,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 19:21:47,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 19:21:47,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 19:21:48,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 19:21:48,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 19:21:49,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 19:21:49,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 19:21:50,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 19:21:50,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 19:21:51,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 19:21:51,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 19:21:52,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 19:21:52,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 19:21:52,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 19:21:53,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 19:21:53,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 19:21:53,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 19:21:54,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 19:21:54,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 19:21:55,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 19:21:55,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 19:21:56,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 19:21:56,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 19:21:57,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 19:21:57,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 19:21:58,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 19:21:58,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 19:21:58,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 19:21:59,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 19:21:59,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 19:22:00,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 19:22:00,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 19:22:01,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 19:22:01,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 19:22:02,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 19:22:02,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 19:22:02,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 19:22:03,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 19:22:03,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 19:22:04,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 19:22:04,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 19:22:05,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 19:22:05,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 19:22:06,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 19:22:06,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 19:22:06,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 19:22:07,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 19:22:07,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 19:22:08,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 19:22:08,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 19:22:08,907 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 19:22:08,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:22:09,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:09,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:22:09,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:09,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:22:09,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:22:09,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:22:09,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:22:09,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:22:09,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:22:09,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:22:09,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 19:22:09,602 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-01 19:22:09,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 19:22:10,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 19:22:10,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 19:22:11,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 19:22:11,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 19:22:12,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 19:22:12,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 19:22:12,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 19:22:13,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 19:22:13,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 19:22:14,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 19:22:14,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 19:22:15,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 19:22:15,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 19:22:16,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 19:22:16,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 19:22:17,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 19:22:17,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 19:22:17,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 19:22:18,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 19:22:18,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 19:22:19,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 19:22:19,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 19:22:20,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 19:22:20,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 19:22:21,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 19:22:21,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 19:22:21,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 19:22:22,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 19:22:22,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 19:22:23,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 19:22:23,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 19:22:24,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 19:22:24,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 19:22:25,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 19:22:25,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 19:22:26,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 19:22:26,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 19:22:27,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 19:22:27,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 19:22:28,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 19:22:28,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 19:22:29,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 19:22:29,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 19:22:30,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 19:22:30,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 19:22:31,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 19:22:31,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 19:22:32,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 19:22:32,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 19:22:33,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 19:22:33,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 19:22:34,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 19:22:34,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 19:22:34,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 19:22:35,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 19:22:35,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 19:22:36,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 19:22:36,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 19:22:37,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 19:22:37,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 19:22:37,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 19:22:38,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 19:22:38,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 19:22:39,477 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:22:39,479 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 47906.734375, u'ReplayWALOffset': u'5248:5816320', u'NTablesCreated': 419931, u'RecentReadIOAmpSet': 0.03576575219631195, u'ReadIOAmp': 520.5797119140625, u'BlockCacheMisses': 22904230, u'NFlushes': 91539, u'RecentWriteAmp': 3.2770161628723145, u'NumSyncFlushes': 3, u'TotalMemUsed': 131479238, u'NBloomFilterHits': 481165759, u'Fragmentation': 0.43051257729530334, u'NumNormalFlushes': 47905, u'RecentBlockCacheHitRatio': 87.56389617919922, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 228680, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 8217624, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 261325203, u'CheckpointOverhead': 8101888, u'NItemsGet': 0, u'NWriteIOs': 331171, u'NReadBytes': 17770993519L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 15255489, u'SizeRatio': 0.33065381917086517, u'NWriteBytes': 730640384, u'BloomFilterMemUsed': 0, u'NumTables': 44, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 569344}, {u'TargetSize': 46137344, u'LogicalSize': 28456156, u'SizeRatio': 0.6167705709284003, u'NWriteBytes': 580325376, u'BloomFilterMemUsed': 0, u'NumTables': 19, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 372736}, {u'TargetSize': 340547964, u'LogicalSize': 282076328, u'SizeRatio': 0.8283013196931049, u'NWriteBytes': 1717956608, u'BloomFilterMemUsed': 0, u'NumTables': 144, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 3395584}, {u'TargetSize': 17375707354L, u'LogicalSize': 15265128576L, u'SizeRatio': 0.8785327851695125, u'NWriteBytes': 3577847808L, u'BloomFilterMemUsed': 0, u'NumTables': 423, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 150401024}, {u'TargetSize': 68109593842L, u'LogicalSize': 34751414709L, u'SizeRatio': 0.5102278952010199, u'NWriteBytes': 47278526464L, u'BloomFilterMemUsed': 0, u'NumTables': 179, u'WastedSpace': 696617788, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 34751414272L}], u'NItemsGarbageCollected': 11061212, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 210690155, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 105792, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6041725, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 310764, u'CompactWriter': {u'NDataBlockWrites': 2089270, u'NWriteBytes': 6157361152L, u'NIndexBlockWrites': 111537, u'NBytesIncoming': 8511254363L, u'NSyncs': 0, u'NInserts': 360366763, u'NWriteIOs': 105309}, u'NWriteBytes': 53837296895L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 11061212, u'LogicalDiskSize': 34751414709L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8651284007004029, u'BufferMemUsed': 494628, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 19486286133L, u'NTableFiles': 1047, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 273296, u'TreeSnapshotMemoryUsed': 126832, u'TotalDiskUsage': 43234770944L, u'BytesIncoming': 62230412100L, u'NItemsDel': 0, u'PendingCheckpointBytes': 10399744, u'NCompacts': 31523, u'NItemsPut': 64277950, u'NSyncs': 43634, u'CompactReader': {u'NReadIOs': 6041725, u'NReadBytes': 17770993519L, u'NDataBlockReads': 5866300, u'NBytesOutgoing': 0, u'NIndexBlockReads': 175425}, u'ActiveDiskUsage': 34906152960L, u'NTablesDeleted': 142454, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 14448562, u'NWriteBytes': 47679935743L, u'NIndexBlockWrites': 358361, u'NBytesIncoming': 62801585005L, u'NSyncs': 43634, u'NInserts': 64268024, u'NWriteIOs': 225862}}, u'NReadBytes': 13640082593033L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 155.6999969482422, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 224081358, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5077, u'NTxns': 4470174, u'NWriteBytes': 75252957184L, u'NSegments': 5249, u'NSyncs': 4470174, u'NLoggedBytes': 75252957184L, u'NActiveSegments': 2, u'NWrites': 4483983, u'NInserts': 82593568, u'DiskUsed': 23699456}, u'NGets': 9742505, u'NReadBytesSet': 30877069083L, u'NReadBytesCompact': 46036803974L, u'IndexResidentRatio': 0.09143517166376114, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 77.49496459960938, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 13594045789059L, u'BloomFilterMemUsedRatio': 0.14129073917865753, u'NWriteBytes': 160451486120L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.5035603046417236, u'NBlocksCached': 33831, u'LogicalDataSize': 20287093638L, u'ActiveBloomFilterMemUsed': 20867944, u'NFileCountCompacts': 10143, u'NTableFiles': 1829, u'FlushQueueSize': 72, u'TableObjectMemUsed': 467232, u'BlockCacheQuota': 101565710, u'TotalDiskUsage': 44239785984L, u'NPurgerRuns': 88, u'ReadIOAmpSet': 0.0, u'NSyncs': 130902, u'NReadIOsSetPerSec': 9.100000381469727, u'NReadIOsPerSec': 184.1999969482422, u'NTablesDeleted': 333207, u'BloomFilterFPR': 0.015378240495920181, u'NBloomFilterHitsPerSec': 1782.8333740234375, u'WriteCacheMemUsed': 10857544, u'RecentBloomFilterFPR': 0.015281230211257935, u'BytesPerRead': 1392164.5, u'BytesIncomingPerSec': 440113.0, u'NReadIOsGetPerSec': 17.96666717529297, u'NGetStats': 339347, u'TotalIndexBlocksSize': 275045697, u'NGetStatsPerSec': 4.433333396911621, u'NWriteBytesCompact': 33330405376L, u'NReadIOsSet': 11643914, u'ActiveDataSize': 20287093638L, u'NWriteBytesCompactPerSec': 429397.34375, u'NBloomFilterMissesPerSec': 27.66666603088379, u'NCommitBatches': 4470174, u'NGetStatsComputed': 339347, u'NWriteIOs': 5140627, u'NReadIOsGet': 5083394474L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 25391428, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 254.43333435058594, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 180864, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 97104825, u'NWriterCompacts': 0, u'NReadIOs': 5100082462L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 649995140, u'NTablesCreated': 100137, u'ReadIOAmp': 96.65209919144826, u'MemtablesMemUsed': 549912, u'NBloomFilterHits': 481165759, u'TotalIndexBlocksSize': 13708062, u'CheckpointOverhead': 14917632, u'NItemsGet': 52594655, u'NWriteIOs': 190725, u'NReadBytes': 13622206328686L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 1998262, u'SizeRatio': 0.04331116242842241, u'NWriteBytes': 3400388608L, u'BloomFilterMemUsed': 28088, u'NumTables': 73, u'WastedSpace': 161, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1904640}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 30332386, u'SizeRatio': 0.6574367609891024, u'NWriteBytes': 10886287360L, u'BloomFilterMemUsed': 927456, u'NumTables': 55, u'WastedSpace': 64598, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 30347264}, {u'TargetSize': 87188765, u'LogicalSize': 71205551, u'SizeRatio': 0.8166826425400108, u'NWriteBytes': 7347896320L, u'BloomFilterMemUsed': 2180928, u'NumTables': 38, u'WastedSpace': 192450, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 71389184}, {u'TargetSize': 871887702, u'LogicalSize': 871887702, u'SizeRatio': 1.0, u'NWriteBytes': 9190649856L, u'BloomFilterMemUsed': 17730064, u'NumTables': 233, u'WastedSpace': 2875084, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 874749952}], u'NItemsGarbageCollected': 18932749, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 13386319, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 50688, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5093848045L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 160457, u'CompactWriter': {u'NDataBlockWrites': 9900458, u'NWriteBytes': 27131617280L, u'NIndexBlockWrites': 129506, u'NBytesIncoming': 37699651870L, u'NSyncs': 0, u'NInserts': 649995140, u'NWriteIOs': 60933}, u'NWriteBytes': 30592625205L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5083383812L, u'NReadBytes': 13594030508437L, u'NDataBlockReads': 11512729332L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 22305008044L}, u'NBloomFilterMisses': 7515051, u'NItemsMerged': 0, u'LogicalDiskSize': 871887702, u'TotalBloomFilterMemUsed': 21069392, u'WriteAmp': 14.569633205835773, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 20866536, u'LogicalDataSize': 800682151, u'NTableFiles': 528, u'NFileCountCompacts': 5086, u'TableObjectMemUsed': 130944, u'TreeSnapshotMemoryUsed': 90384, u'TotalDiskUsage': 1002934272, u'BytesIncoming': 2099752600, u'NItemsDel': 0, u'PendingCheckpointBytes': 23052288, u'NCompacts': 6530, u'NItemsPut': 42852150, u'NSyncs': 43634, u'CompactReader': {u'NReadIOs': 10464233, u'NReadBytes': 28175820249L, u'NDataBlockReads': 10252844, u'NBytesOutgoing': 0, u'NIndexBlockReads': 211389}, u'ActiveDiskUsage': 978391040, u'NTablesDeleted': 99738, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015378240451062525, u'Writer': {u'NDataBlockWrites': 715447, u'NWriteBytes': 3461007925L, u'NIndexBlockWrites': 86392, u'NBytesIncoming': 2485134096L, u'NSyncs': 43634, u'NInserts': 42847187, u'NWriteIOs': 129792}}, u'NReadBytesCompactPerSec': 425007.4375, u'TableMetaMemUsed': 544105, u'NDeletes': 0, u'BlockCacheHitsPerSec': 1096.300048828125, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 7515051, u'LogicalDiskSize': 35623427765L, u'NSets': 42852150, u'TotalBloomFilterMemUsed': 21070800, u'BufferMemUsed': 494628, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 140.5, u'FragmentationWithOverhead': 0.541428804397583, u'NCompactsPerSec': 0.36666667461395264, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 472914.15625, u'TreeSnapshotMemoryUsed': 262048, u'BytesIncoming': 64089325577L, u'NCompacts': 43110, u'NumPersistentFlushes': 43631, u'NWriteIOsPerSec': 146.73333740234375, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 35885355008L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 4.433333396911621, u'NReadBytesSetPerSec': 25057.400390625, u'BlockCacheHits': 78869595, u'NWriteBytesPerSec': 1442257.5, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 91114, u'ReadIOAmp': 0.0011907620452695073, u'MemtablesMemUsed': 2090008, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 12432, u'CheckpointOverhead': 1269760, u'NItemsGet': 8953930, u'NWriteIOs': 134748, u'NReadBytes': 105270828, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 800610, u'SizeRatio': 0.01735275441950018, u'NWriteBytes': 705024000, u'BloomFilterMemUsed': 704, u'NumTables': 88, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 720896}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125354, u'SizeRatio': 0.0027169747786088424, u'NWriteBytes': 41426944, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 26323356, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4884, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 24384, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 192692, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 72884, u'CompactWriter': {u'NDataBlockWrites': 5057, u'NWriteBytes': 41426944, u'NIndexBlockWrites': 5057, u'NBytesIncoming': 22725986, u'NSyncs': 0, u'NInserts': 268021, u'NWriteIOs': 5057}, u'NWriteBytes': 768606836, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 10662, u'NReadBytes': 15280622, u'NDataBlockReads': 4458555, u'NBytesOutgoing': 0, u'NIndexBlockReads': 4458555}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125354, u'TotalBloomFilterMemUsed': 1408, u'WriteAmp': 0.2898571392134617, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1408, u'LogicalDataSize': 125354, u'NTableFiles': 254, u'NFileCountCompacts': 5057, u'TableObjectMemUsed': 62992, u'TreeSnapshotMemoryUsed': 44832, u'TotalDiskUsage': 2080768, u'BytesIncoming': 2651674677L, u'NItemsDel': 0, u'PendingCheckpointBytes': 8192, u'NCompacts': 5057, u'NItemsPut': 26330910, u'NSyncs': 43634, u'CompactReader': {u'NReadIOs': 182030, u'NReadBytes': 89990206, u'NDataBlockReads': 91015, u'NBytesOutgoing': 0, u'NIndexBlockReads': 91015}, u'ActiveDiskUsage': 811008, u'NTablesDeleted': 91015, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 86057, u'NWriteBytes': 727179892, u'NIndexBlockWrites': 86057, u'NBytesIncoming': 64035988, u'NSyncs': 43634, u'NInserts': 840012, u'NWriteIOs': 129691}}}}} 2021-12-01 19:22:39,483 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.43051257729530334, 0.43357327580451965, 0.43613389134407043, 0.43575453758239746, 0.4345661699771881, 0.4387292265892029, 0.4372095763683319, 0.4370124936103821, 0.4363447427749634, 0.44276300072669983, 0.43908655643463135, 0.4404274821281433, 0.4369151294231415, 0.43762993812561035, 0.4367634057998657, 0.43644022941589355, 0.4365817904472351, 0.43936580419540405, 0.4381030797958374, 0.43975916504859924, 0.4435546100139618, 0.4348592758178711, 0.4412572681903839, 0.4429730176925659, 0.4431270658969879, 0.4389512836933136, 0.4368695914745331, 0.4399797320365906, 0.4333602786064148, 0.43938684463500977, 0.4379119277000427, 0.4402140974998474, 0.4381754696369171, 0.4393698573112488, 0.44125017523765564, 0.438884973526001, 0.4328172504901886, 0.4357966482639313, 0.4359534978866577, 0.43418270349502563, 0.44299668073654175, 0.43378716707229614, 0.442330002784729, 0.4391922354698181, 0.4405982494354248, 0.4362667202949524, 0.4307308495044708, 0.43473389744758606, 0.4418933391571045, 0.43144240975379944, 0.43222975730895996, 0.4413243532180786, 0.43988099694252014, 0.4283526837825775, 0.43863195180892944, 0.43971705436706543, 0.4427528977394104, 0.43757179379463196, 0.43876951932907104, 0.44023430347442627, 0.43525466322898865, 0.44101202487945557, 0.4313848912715912, 0.4373176395893097], '172.23.110.66': [0.44133260846138, 0.4383646249771118, 0.437911719083786, 0.4341670870780945, 0.4398013651371002, 0.44525137543678284, 0.43796584010124207, 0.4341973066329956, 0.43765997886657715, 0.4392167329788208, 0.4394957721233368, 0.43842244148254395, 0.44327348470687866, 0.4317106604576111, 0.4414966404438019, 0.4377022683620453, 0.43748119473457336, 0.43885907530784607, 0.4423665702342987, 0.44499701261520386, 0.4382037818431854, 0.44266369938850403, 0.4407753050327301, 0.4421333074569702, 0.44189757108688354, 0.4367196559906006, 0.43921327590942383, 0.44119828939437866, 0.4354446828365326, 0.44128289818763733, 0.44319695234298706, 0.44150862097740173, 0.43727293610572815, 0.43903034925460815, 0.4416823387145996, 0.43441319465637207, 0.44053012132644653, 0.43915995955467224, 0.43933239579200745, 0.43686383962631226, 0.43805739283561707, 0.43793246150016785, 0.44866296648979187, 0.43387219309806824, 0.4371073544025421, 0.4381583333015442, 0.4375433921813965, 0.4348268210887909, 0.43821296095848083, 0.436965674161911, 0.43966948986053467, 0.4378790259361267, 0.4390581250190735, 0.43316709995269775, 0.4471888542175293, 0.43844708800315857, 0.43560728430747986, 0.44247400760650635, 0.4433111548423767, 0.43429574370384216, 0.43764278292655945, 0.43802982568740845, 0.4360767900943756, 0.443398654460907], '172.23.110.64': [0.4425680637359619, 0.4384634792804718, 0.4387998878955841, 0.4381934702396393, 0.43773338198661804, 0.43735432624816895, 0.43807944655418396, 0.43316420912742615, 0.4343743622303009, 0.4410208463668823, 0.43531927466392517, 0.43669748306274414, 0.4354046583175659, 0.4395199418067932, 0.44063326716423035, 0.4382663369178772, 0.44009634852409363, 0.43505358695983887, 0.4335850179195404, 0.4421083629131317, 0.4333435893058777, 0.4335172474384308, 0.4299597144126892, 0.44161802530288696, 0.4397576153278351, 0.43945690989494324, 0.4297201335430145, 0.43620970845222473, 0.43756574392318726, 0.43667304515838623, 0.4355957806110382, 0.43747928738594055, 0.43286100029945374, 0.4392947554588318, 0.4400483965873718, 0.43603432178497314, 0.4382724165916443, 0.43524783849716187, 0.44096076488494873, 0.4423859417438507, 0.4382009208202362, 0.4351942837238312, 0.4413354992866516, 0.43847641348838806, 0.4403814375400543, 0.4427471458911896, 0.4446125030517578, 0.4393671154975891, 0.436768501996994, 0.43676039576530457, 0.4368792176246643, 0.4372175633907318, 0.43858471512794495, 0.43893638253211975, 0.43542805314064026, 0.4354481101036072, 0.43655046820640564, 0.43896380066871643, 0.4320640563964844, 0.44015705585479736, 0.4408159554004669, 0.4325871765613556, 0.44186997413635254, 0.4369569420814514]} 2021-12-01 19:22:39,851 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 44.0 2021-12-01 19:22:40,180 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 44.0 2021-12-01 19:22:40,509 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 44.0 2021-12-01 19:22:40,509 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 44.0, '172.23.110.66': 44.0, '172.23.110.64': 44.0} 2021-12-01 19:22:40,513 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:22:40,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:40,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:22:40,994 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:40,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:22:41,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:22:41,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:22:41,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:22:41,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:22:41,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:22:41,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:22:41,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 19:22:41,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-01 19:22:41,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 19:22:41,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:41,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:41,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:41,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:41,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:41,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:41,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:41,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:41,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 19:22:41,538 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-01 19:22:41,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 19:22:41,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:41,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 19:22:41,579 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-01 19:22:41,579 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:22:41,584 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:22:41,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:41,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:22:42,055 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:42,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:22:42,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:22:42,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:22:42,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:22:42,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:22:42,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:22:42,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:22:42,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 19:22:42,196 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-01 19:22:42,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 19:22:42,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:42,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:42,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:42,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:42,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:42,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:42,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:42,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:42,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 19:22:42,545 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-01 19:22:42,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 19:22:42,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:42,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 19:22:42,589 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-01 19:22:42,591 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:22:42,594 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:22:42,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:43,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:22:43,101 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:43,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:22:43,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:22:43,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:22:43,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:22:43,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:22:43,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:22:43,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:22:43,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 19:22:43,256 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-01 19:22:43,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 19:22:43,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:43,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:43,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:43,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:43,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:43,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:43,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:43,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:43,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 19:22:43,585 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-01 19:22:43,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 19:22:43,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:43,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 19:22:43,638 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-01 19:22:43,638 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 3: Create 40000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 7300.03125 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 40000000 Create End: 80000000 Final Start: 40000000 Final End: 80000000 2021-12-01 19:22:43,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:43,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638415364.7 2021-12-01 19:22:44,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:44,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638415365.71 2021-12-01 19:22:45,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:45,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638415366.72 2021-12-01 19:22:46,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:46,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638415367.73 2021-12-01 19:22:47,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:47,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638415368.74 2021-12-01 19:22:48,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:48,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638415369.76 2021-12-01 19:22:49,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:49,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638415370.79 2021-12-01 19:22:50,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:50,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638415371.82 2021-12-01 19:22:51,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:51,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638415372.83 2021-12-01 19:22:52,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:52,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638415373.84 2021-12-01 19:22:53,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:53,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:54,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:54,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:55,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:55,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:56,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:56,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:57,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:57,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:58,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:58,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:59,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:59,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:00,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:00,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:01,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:01,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:02,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:02,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:04,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:04,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:05,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:05,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:06,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:06,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:07,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:07,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:08,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:08,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:09,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:09,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:10,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:10,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:11,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:11,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:12,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:12,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:13,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:13,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:14,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:14,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:15,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:15,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:16,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:16,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:17,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:17,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:18,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:18,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:19,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:19,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:20,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:20,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:21,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:21,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:22,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:22,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:23,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:23,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:24,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:24,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:25,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:25,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:26,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:26,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:27,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:27,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:28,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:28,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:29,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:29,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:30,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:30,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:31,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:31,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:32,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:32,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:33,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:33,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection40_2_1638415372.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638415374.85 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638415364.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638415375.87 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638415368.74 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638415376.88 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638415370.79 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638415377.89 Loader_GleamBookUsers0__default_VolumeCollection44_8_1638415366.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638415378.9 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638415365.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638415379.95 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638415373.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638415380.96 Loader_GleamBookUsers0__default_VolumeCollection49_5_1638415369.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638415381.97 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638415367.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638415382.98 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638415371.82 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638415384.0 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638415374.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638415385.01 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638415375.87 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638415386.07 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638415376.88 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638415387.08 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638415377.89 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638415388.1 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638415379.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638415389.12 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638415384.0 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638415390.13 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638415381.97 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638415391.15 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638415378.9 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638415392.16 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638415380.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638415393.18 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638415386.07 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638415394.19 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638415382.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638415395.21 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638415388.1 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638415396.22 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638415385.01 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638415397.23 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638415390.13 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638415398.25 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638415387.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638415399.26 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638415392.16 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638415400.28 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638415389.12 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638415401.29 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638415394.19 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638415402.31 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638415391.15 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638415403.32 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638415398.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638415404.33 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638415395.21 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638415405.35 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638415393.18 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638415406.36 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638415396.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638415407.37 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638415400.28 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638415408.38 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638415402.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638415409.4 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638415397.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638415410.41 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638415399.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638415411.42 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638415401.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638415412.43 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638415404.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638415413.45 Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638415406.36 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638415414.56 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638415403.32 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638415410.41 is completed! Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638415405.35 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638415407.37 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638415408.38 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638415409.4 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638415411.42 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638415414.56 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638415412.43 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638415413.45 is completed! 2021-12-01 23:28:02,736 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 23:28:02,740 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:28:02,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:03,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:28:03,209 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:03,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:28:03,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:28:03,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:28:03,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:28:03,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:28:03,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:28:03,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:28:03,355 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 23:28:03,355 | infra | INFO | pool-3-thread-14 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36' started 2021-12-01 23:28:03,355 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36 2021-12-01 23:28:03,358 | infra | DEBUG | pool-3-thread-14 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 23:28:03,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:28:03,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:03,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:28:03,801 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:03,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:28:03,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:28:03,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:28:03,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:28:03,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:28:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:28:03,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:28:03,903 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 23:28:03,903 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9 2021-12-01 23:28:03,904 | infra | INFO | pool-3-thread-15 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9' started 2021-12-01 23:28:03,907 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 23:28:03,910 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:28:04,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:04,092 | test | DEBUG | pool-3-thread-14 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 23:28:04,092 | test | DEBUG | pool-3-thread-14 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-01 23:28:04,092 | infra | INFO | pool-3-thread-14 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36' completed 2021-12-01 23:28:04,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:28:04,408 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:04,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:28:04,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:28:04,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:28:04,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:28:04,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:28:04,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:28:04,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:28:04,582 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 23:28:04,582 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58 2021-12-01 23:28:04,584 | infra | INFO | pool-3-thread-16 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58' started 2021-12-01 23:28:04,585 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36 2021-12-01 23:28:04,585 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:28:04,585 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 23:28:04,585 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9 2021-12-01 23:28:04,690 | test | DEBUG | pool-3-thread-15 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 23:28:04,690 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-01 23:28:04,690 | infra | INFO | pool-3-thread-15 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9' completed 2021-12-01 23:28:04,691 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:28:04,691 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58 2021-12-01 23:28:05,372 | test | DEBUG | pool-3-thread-16 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 23:28:05,372 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-01 23:28:05,374 | infra | INFO | pool-3-thread-16 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58' completed 2021-12-01 23:28:05,375 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 23:28:05,415 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 19340.7, u'diskFetches': 0, u'dataUsed': 8173590473363L, u'itemCount': 3999973938L, u'quotaPercentUsed': 79.26818212121725, u'storageTotals': {u'hdd': {u'usedByData': 11318864466026L, u'used': 11432856681180L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 24542953309476L}, u'ram': {u'usedByData': 255340687368L, u'used': 375047712768L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 11318864466026L, u'vbActiveNumNonResident': 3866172293L, u'memUsed': 255340687368L} 2021-12-01 23:28:05,766 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 3999973938 | 300.00 GiB | 237.80 GiB | 10.29 TiB | 3.34506291976 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-01 23:28:05,766 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+-------------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+-------------------+---------------+---------------+ | 2000000000 | 4000000000 | 0-0 | 40000000-80000000 | 0-0 | 0-0 | +---------------+---------------+---------------+-------------------+---------------+---------------+ 2021-12-01 23:28:06,128 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.34506291976 2021-12-01 23:28:06,131 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:28:06,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:06,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:28:06,592 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:06,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:28:06,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:28:06,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:28:06,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:28:06,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:28:06,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:28:06,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:28:06,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 23:28:09,232 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:28:09,236 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:28:09,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:09,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:28:09,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:09,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:28:09,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:28:09,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:28:09,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:28:09,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:28:09,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:28:09,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:28:09,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 23:28:12,357 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:28:12,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:28:12,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:12,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:28:12,852 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:12,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:28:12,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:28:12,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:28:12,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:28:12,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:28:12,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:28:12,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:28:13,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 23:28:15,598 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 23:28:15,598 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 10834726MB 2021-12-01 23:28:15,599 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 7824MB 2021-12-01 23:28:15,601 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 351884MB 2021-12-01 23:28:15,602 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 10482432MB 2021-12-01 23:28:15,605 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:28:15,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:15,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:28:16,076 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:16,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:28:16,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:28:16,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:28:16,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:28:16,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:28:16,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:28:16,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:28:16,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 23:28:16,282 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-01 23:28:16,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 23:28:16,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 23:28:17,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 23:28:17,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 23:28:17,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 23:28:18,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 23:28:18,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 23:28:18,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 23:28:19,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 23:28:19,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 23:28:20,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 23:28:20,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 23:28:21,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 23:28:21,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 23:28:22,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 23:28:22,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 23:28:23,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 23:28:23,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 23:28:23,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 23:28:24,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 23:28:24,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 23:28:25,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 23:28:25,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 23:28:26,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 23:28:26,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 23:28:27,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 23:28:27,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 23:28:28,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 23:28:28,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 23:28:29,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 23:28:29,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 23:28:29,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 23:28:30,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 23:28:30,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 23:28:31,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 23:28:31,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 23:28:32,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 23:28:32,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 23:28:33,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 23:28:33,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 23:28:34,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 23:28:34,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 23:28:34,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 23:28:35,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 23:28:35,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 23:28:36,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 23:28:36,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 23:28:37,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 23:28:37,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 23:28:37,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 23:28:38,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 23:28:38,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 23:28:39,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 23:28:39,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 23:28:39,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 23:28:40,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 23:28:40,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 23:28:41,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 23:28:41,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 23:28:42,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 23:28:42,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 23:28:42,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 23:28:43,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 23:28:43,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 23:28:43,755 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:28:43,757 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:28:43,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:44,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:28:44,236 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:44,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:28:44,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:28:44,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:28:44,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:28:44,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:28:44,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:28:44,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:28:44,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 23:28:44,441 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-01 23:28:44,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 23:28:44,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 23:28:45,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 23:28:45,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 23:28:46,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 23:28:46,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 23:28:46,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 23:28:47,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 23:28:47,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 23:28:48,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 23:28:48,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 23:28:48,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 23:28:49,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 23:28:49,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 23:28:50,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 23:28:50,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 23:28:50,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 23:28:51,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 23:28:51,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 23:28:52,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 23:28:52,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 23:28:53,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 23:28:53,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 23:28:53,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 23:28:54,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 23:28:54,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 23:28:55,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 23:28:55,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 23:28:56,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 23:28:56,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 23:28:56,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 23:28:57,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 23:28:57,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 23:28:58,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 23:28:58,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 23:28:59,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 23:28:59,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 23:28:59,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 23:29:00,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 23:29:00,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 23:29:01,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 23:29:01,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 23:29:02,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 23:29:02,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 23:29:03,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 23:29:03,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 23:29:03,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 23:29:04,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 23:29:04,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 23:29:05,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 23:29:05,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 23:29:06,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 23:29:06,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 23:29:07,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 23:29:07,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 23:29:08,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 23:29:08,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 23:29:09,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 23:29:09,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 23:29:10,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 23:29:10,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 23:29:11,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 23:29:11,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 23:29:11,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 23:29:12,289 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 23:29:12,293 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:29:12,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:12,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:29:12,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:12,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:29:12,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:29:12,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:29:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:29:12,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:29:12,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:29:12,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:29:12,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 23:29:13,036 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-01 23:29:13,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 23:29:13,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 23:29:13,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 23:29:14,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 23:29:14,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 23:29:15,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 23:29:15,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 23:29:16,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 23:29:16,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 23:29:16,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 23:29:17,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 23:29:17,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 23:29:18,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 23:29:18,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 23:29:19,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 23:29:19,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 23:29:20,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 23:29:20,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 23:29:21,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 23:29:21,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 23:29:22,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 23:29:22,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 23:29:22,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 23:29:23,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 23:29:23,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 23:29:24,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 23:29:24,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 23:29:24,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 23:29:25,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 23:29:25,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 23:29:26,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 23:29:26,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 23:29:27,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 23:29:27,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 23:29:28,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 23:29:28,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 23:29:29,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 23:29:29,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 23:29:30,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 23:29:30,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 23:29:30,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 23:29:31,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 23:29:31,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 23:29:32,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 23:29:32,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 23:29:33,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 23:29:33,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 23:29:34,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 23:29:34,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 23:29:35,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 23:29:35,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 23:29:35,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 23:29:36,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 23:29:36,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 23:29:37,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 23:29:37,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 23:29:38,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 23:29:38,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 23:29:39,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 23:29:39,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 23:29:40,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 23:29:40,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 23:29:41,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 23:29:41,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 23:29:42,012 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:29:42,013 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 27898.5, u'ReplayWALOffset': u'7147:9367552', u'NTablesCreated': 607405, u'RecentReadIOAmpSet': 0.022602416574954987, u'ReadIOAmp': 260.45208740234375, u'BlockCacheMisses': 32195354, u'NFlushes': 152017, u'RecentWriteAmp': 4.346617698669434, u'NumSyncFlushes': 1190, u'TotalMemUsed': 128207800, u'NBloomFilterHits': 789112675, u'Fragmentation': 0.26717230677604675, u'NumNormalFlushes': 78947, u'RecentBlockCacheHitRatio': 62.07989501953125, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 286926, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 6153240, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 403954899, u'CheckpointOverhead': 0, u'NItemsGet': 0, u'NWriteIOs': 449880, u'NReadBytes': 17770993519L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 22390577, u'SizeRatio': 0.4853026866912842, u'NWriteBytes': 730820608, u'BloomFilterMemUsed': 0, u'NumTables': 55, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 749568}, {u'TargetSize': 46137344, u'LogicalSize': 28456156, u'SizeRatio': 0.6167705709284003, u'NWriteBytes': 580325376, u'BloomFilterMemUsed': 0, u'NumTables': 19, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 372736}, {u'TargetSize': 570541359, u'LogicalSize': 282076328, u'SizeRatio': 0.4944011920439934, u'NWriteBytes': 1717956608, u'BloomFilterMemUsed': 0, u'NumTables': 144, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 3395584}, {u'TargetSize': 29184532698L, u'LogicalSize': 15883439486L, u'SizeRatio': 0.5442416930351769, u'NWriteBytes': 3577847808L, u'BloomFilterMemUsed': 0, u'NumTables': 423, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 150401024}, {u'TargetSize': 114108273398L, u'LogicalSize': 58369065397L, u'SizeRatio': 0.5115235176104509, u'NWriteBytes': 70896177152L, u'BloomFilterMemUsed': 0, u'NumTables': 242, u'WastedSpace': 1314928698, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 58369064960L}], u'NItemsGarbageCollected': 11061212, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 351333270, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 89760, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6041725, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 263670, u'CompactWriter': {u'NDataBlockWrites': 2089270, u'NWriteBytes': 6157361152L, u'NIndexBlockWrites': 111537, u'NBytesIncoming': 8511254363L, u'NSyncs': 0, u'NInserts': 360366763, u'NWriteIOs': 105309}, u'NWriteBytes': 77536309472L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 11061212, u'LogicalDiskSize': 58369065397L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8314528017855712, u'BufferMemUsed': 479590, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 42485625911L, u'NTableFiles': 883, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 231880, u'TreeSnapshotMemoryUsed': 132208, u'TotalDiskUsage': 67372199936L, u'BytesIncoming': 93254011900L, u'NItemsDel': 0, u'PendingCheckpointBytes': 33366016, u'NCompacts': 31523, u'NItemsPut': 85736900, u'NSyncs': 73070, u'CompactReader': {u'NReadIOs': 6041725, u'NReadBytes': 17770993519L, u'NDataBlockReads': 5866300, u'NBytesOutgoing': 0, u'NIndexBlockReads': 175425}, u'ActiveDiskUsage': 58523983872L, u'NTablesDeleted': 142454, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 21621202, u'NWriteBytes': 71378948320L, u'NIndexBlockWrites': 575766, u'NBytesIncoming': 94019732811L, u'NSyncs': 73070, u'NInserts': 85732837, u'NWriteIOs': 344571}}, u'NReadBytes': 13679900840289L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 303.1333312988281, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 375320295, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5068, u'NTxns': 6814901, u'NWriteBytes': 113121427456L, u'NSegments': 7149, u'NSyncs': 6814901, u'NLoggedBytes': 113121427456L, u'NActiveSegments': 3, u'NWrites': 6829309, u'NInserts': 125310351, u'DiskUsed': 42422272}, u'NGets': 19497058, u'NReadBytesSet': 33395671743L, u'NReadBytesCompact': 65993295453L, u'IndexResidentRatio': 0.04219229519367218, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.8999999761581421, u'BlockCacheHitRatio': 74.92744445800781, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 13613907544836L, u'BloomFilterMemUsedRatio': 0.26079338788986206, u'NWriteBytes': 244213584772L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.114255391061306, u'WriteAmp': 2.5368237495422363, u'NBlocksCached': 27239, u'LogicalDataSize': 43933039214L, u'ActiveBloomFilterMemUsed': 38306200, u'NFileCountCompacts': 17007, u'NTableFiles': 1925, u'FlushQueueSize': 72, u'TableObjectMemUsed': 490296, u'BlockCacheQuota': 87296067, u'TotalDiskUsage': 69185826816L, u'NPurgerRuns': 132, u'ReadIOAmpSet': 0.0, u'NSyncs': 219210, u'NReadIOsSetPerSec': 4.800000190734863, u'NReadIOsPerSec': 304.3666687011719, u'NTablesDeleted': 462290, u'BloomFilterFPR': 0.01651875674724579, u'NBloomFilterHitsPerSec': 2802.300048828125, u'WriteCacheMemUsed': 8694856, u'RecentBloomFilterFPR': 0.02165716327726841, u'BytesPerRead': 696541.5625, u'BytesIncomingPerSec': 398697.6875, u'NReadIOsGetPerSec': 10.766666412353516, u'NGetStats': 405975, u'TotalIndexBlocksSize': 428770210, u'NGetStatsPerSec': 4.566666603088379, u'NWriteBytesCompact': 53178306560L, u'NReadIOsSet': 12563943, u'ActiveDataSize': 43933039214L, u'NWriteBytesCompactPerSec': 769228.8125, u'NBloomFilterMissesPerSec': 62.03333282470703, u'NCommitBatches': 6814901, u'NGetStatsComputed': 405975, u'NWriteIOs': 7827508, u'NReadIOsGet': 5090613189L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 21824018, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 212.36666870117188, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.6666666865348816, u'ManagedFileObjectMemUsed': 189792, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 78108173, u'NWriterCompacts': 0, u'NReadIOs': 5114907704L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1126593024, u'NTablesCreated': 167704, u'ReadIOAmp': 60.74105183173218, u'MemtablesMemUsed': 369688, u'NBloomFilterHits': 789112675, u'TotalIndexBlocksSize': 24799891, u'CheckpointOverhead': 32198656, u'NItemsGet': 83808158, u'NWriteIOs': 322474, u'NReadBytes': 13661952832536L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2510752, u'SizeRatio': 0.054419084028764206, u'NWriteBytes': 5206331392L, u'BloomFilterMemUsed': 33416, u'NumTables': 100, u'WastedSpace': 28, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2392064}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 27065237, u'SizeRatio': 0.5866232135079124, u'NWriteBytes': 21150195712L, u'BloomFilterMemUsed': 831744, u'NumTables': 73, u'WastedSpace': 42240, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 27004928}, {u'TargetSize': 158083289, u'LogicalSize': 133545146, u'SizeRatio': 0.8447771225205214, u'NWriteBytes': 10362413056L, u'BloomFilterMemUsed': 4079808, u'NumTables': 93, u'WastedSpace': 331323, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 133832704}, {u'TargetSize': 1580832927, u'LogicalSize': 1580832927, u'SizeRatio': 1.0, u'NWriteBytes': 15838908416L, u'BloomFilterMemUsed': 33359880, u'NumTables': 285, u'WastedSpace': 5377605, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1586208768}], u'NItemsGarbageCollected': 21425800, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 23982477, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 69792, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5108543093L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 220953, u'CompactWriter': {u'NDataBlockWrites': 17154940, u'NWriteBytes': 46951452672L, u'NIndexBlockWrites': 223041, u'NBytesIncoming': 65342312392L, u'NSyncs': 0, u'NInserts': 1126593024, u'NWriteIOs': 105011}, u'NWriteBytes': 52266792861L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5090595669L, u'NReadBytes': 13613882079037L, u'NDataBlockReads': 11541843537L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 22348724689L}, u'NBloomFilterMisses': 13254101, u'NItemsMerged': 0, u'LogicalDiskSize': 1580832927, u'TotalBloomFilterMemUsed': 38890576, u'WriteAmp': 16.586097468611694, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 38304848, u'LogicalDataSize': 1447287781, u'NTableFiles': 727, u'NFileCountCompacts': 8524, u'TableObjectMemUsed': 180296, u'TreeSnapshotMemoryUsed': 102832, u'TotalDiskUsage': 1811046400, u'BytesIncoming': 3151241150L, u'NItemsDel': 0, u'PendingCheckpointBytes': 58761216, u'NCompacts': 10783, u'NItemsPut': 64311100, u'NSyncs': 73070, u'CompactReader': {u'NReadIOs': 17947424, u'NReadBytes': 48070753499L, u'NDataBlockReads': 17587689, u'NBytesOutgoing': 0, u'NIndexBlockReads': 359735}, u'ActiveDiskUsage': 1749438464, u'NTablesDeleted': 167153, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.016518756005919167, u'Writer': {u'NDataBlockWrites': 1082982, u'NWriteBytes': 5315340189L, u'NIndexBlockWrites': 144627, u'NBytesIncoming': 3729805396L, u'NSyncs': 73070, u'NInserts': 64307037, u'NWriteIOs': 217463}}, u'NReadBytesCompactPerSec': 755318.6875, u'TableMetaMemUsed': 575013, u'NDeletes': 0, u'BlockCacheHitsPerSec': 496.26666259765625, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 13254101, u'LogicalDiskSize': 59950023846L, u'NSets': 64311100, u'TotalBloomFilterMemUsed': 38892328, u'BufferMemUsed': 479590, u'RecentReadAmp': 0.0, u'NGetsPerSec': 94.23332977294922, u'NInserts': 0, u'NCommitBatchesPerSec': 176.73333740234375, u'FragmentationWithOverhead': 0.3649994134902954, u'NCompactsPerSec': 0.23333333432674408, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 783217.125, u'TreeSnapshotMemoryUsed': 280560, u'BytesIncoming': 96267460842L, u'NCompacts': 50789, u'NumPersistentFlushes': 71880, u'NWriteIOsPerSec': 181.53334045410156, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 296.0576477050781, u'ActiveDiskUsage': 60274176000L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 4.566666603088379, u'NReadBytesSetPerSec': 12506.06640625, u'BlockCacheHits': 96213385, u'NWriteBytesPerSec': 1732986.375, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 152775, u'ReadIOAmp': 0.00128333879045319, u'MemtablesMemUsed': 2171928, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 15420, u'CheckpointOverhead': 1531904, u'NItemsGet': 13651890, u'NWriteIOs': 225845, u'NReadBytes': 177014234, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 733905, u'SizeRatio': 0.01590696248141202, u'NWriteBytes': 1182085120, u'BloomFilterMemUsed': 648, u'NumTables': 81, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 663552}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125522, u'SizeRatio': 0.002720616080544212, u'NWriteBytes': 69492736, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 40547075, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4548, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 30240, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 322886, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 90390, u'CompactWriter': {u'NDataBlockWrites': 8483, u'NWriteBytes': 69492736, u'NIndexBlockWrites': 8483, u'NBytesIncoming': 38505077, u'NSyncs': 0, u'NInserts': 449599, u'NWriteIOs': 8483}, u'NWriteBytes': 1289054983, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 17520, u'NReadBytes': 25465799, u'NDataBlockReads': 6803285, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6803285}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125522, u'TotalBloomFilterMemUsed': 1752, u'WriteAmp': 0.31721077306115586, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1352, u'LogicalDataSize': 125522, u'NTableFiles': 315, u'NFileCountCompacts': 8483, u'TableObjectMemUsed': 78120, u'TreeSnapshotMemoryUsed': 45520, u'TotalDiskUsage': 2580480, u'BytesIncoming': 4063717542L, u'NItemsDel': 0, u'PendingCheckpointBytes': 327680, u'NCompacts': 8483, u'NItemsPut': 40554562, u'NSyncs': 73070, u'CompactReader': {u'NReadIOs': 305366, u'NReadBytes': 151548435, u'NDataBlockReads': 152683, u'NBytesOutgoing': 0, u'NIndexBlockReads': 152683}, u'ActiveDiskUsage': 753664, u'NTablesDeleted': 152683, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 144292, u'NWriteBytes': 1219562247, u'NIndexBlockWrites': 144292, u'NBytesIncoming': 107664417, u'NSyncs': 73070, u'NInserts': 1401980, u'NWriteIOs': 217362}}}}} 2021-12-01 23:29:42,017 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.26717230677604675, 0.26966872811317444, 0.27099689841270447, 0.2712387144565582, 0.2698584198951721, 0.27338850498199463, 0.272088885307312, 0.2725244164466858, 0.27154380083084106, 0.27637338638305664, 0.273430198431015, 0.27471795678138733, 0.2719164490699768, 0.2722247540950775, 0.2720230221748352, 0.2713126838207245, 0.2712197005748749, 0.2732507586479187, 0.2718661427497864, 0.2734212279319763, 0.2755495309829712, 0.2702828645706177, 0.27385225892066956, 0.2758881449699402, 0.2766515016555786, 0.2730448544025421, 0.27125683426856995, 0.27276208996772766, 0.2681541442871094, 0.2728855311870575, 0.2712186276912689, 0.2733805775642395, 0.27213162183761597, 0.27228212356567383, 0.2739677429199219, 0.2715042531490326, 0.26718762516975403, 0.2688038647174835, 0.26919469237327576, 0.2675807774066925, 0.27470335364341736, 0.26790717244148254, 0.2741203308105469, 0.27179351449012756, 0.27285391092300415, 0.2700207233428955, 0.26604071259498596, 0.26855573058128357, 0.27383559942245483, 0.26673799753189087, 0.2671404182910919, 0.27322837710380554, 0.2721586525440216, 0.26348963379859924, 0.2719159424304962, 0.272703617811203, 0.2744605243206024, 0.2713673710823059, 0.27147090435028076, 0.2731764316558838, 0.26921701431274414, 0.2740108072757721, 0.26772400736808777, 0.2717118561267853], '172.23.110.66': [0.27430376410484314, 0.2724369764328003, 0.27189722657203674, 0.2688954770565033, 0.273001104593277, 0.2767811417579651, 0.2713577151298523, 0.2697407901287079, 0.27154281735420227, 0.27364715933799744, 0.2735995948314667, 0.27220192551612854, 0.27659323811531067, 0.26803308725357056, 0.27508237957954407, 0.27201950550079346, 0.2717548608779907, 0.272646427154541, 0.2746811509132385, 0.2770567536354065, 0.2720875144004822, 0.2750941216945648, 0.27348628640174866, 0.27513065934181213, 0.27487021684646606, 0.27031636238098145, 0.27240005135536194, 0.27471283078193665, 0.27051597833633423, 0.27481314539909363, 0.27609238028526306, 0.2749500572681427, 0.272358775138855, 0.27280277013778687, 0.27497169375419617, 0.26955756545066833, 0.2737216353416443, 0.273054838180542, 0.27329757809638977, 0.27127277851104736, 0.27284979820251465, 0.2723167836666107, 0.28024041652679443, 0.2694976031780243, 0.2715397775173187, 0.27267101407051086, 0.2723013460636139, 0.2698461413383484, 0.2727219760417938, 0.2717038094997406, 0.2736717164516449, 0.27237388491630554, 0.27292415499687195, 0.2689853310585022, 0.2787993848323822, 0.27201178669929504, 0.27004969120025635, 0.27535340189933777, 0.275543212890625, 0.26887568831443787, 0.27122870087623596, 0.2718653976917267, 0.2701369822025299, 0.2754764258861542], '172.23.110.64': [0.2751578986644745, 0.2725180983543396, 0.2725996673107147, 0.272032231092453, 0.27160584926605225, 0.2719644606113434, 0.2719409763813019, 0.26884725689888, 0.26918825507164, 0.2740488350391388, 0.26969820261001587, 0.2704392373561859, 0.2699264883995056, 0.2720470130443573, 0.27330055832862854, 0.27126625180244446, 0.27332088351249695, 0.26947546005249023, 0.2683144211769104, 0.27454429864883423, 0.2683924436569214, 0.26846009492874146, 0.2658545672893524, 0.27432525157928467, 0.2725927233695984, 0.2727634012699127, 0.26587268710136414, 0.26990944147109985, 0.27120527625083923, 0.2704262137413025, 0.26946496963500977, 0.2711156904697418, 0.2682006359100342, 0.27195075154304504, 0.27265694737434387, 0.270295649766922, 0.2722350060939789, 0.26966607570648193, 0.27424487471580505, 0.27509450912475586, 0.27275171875953674, 0.2696770429611206, 0.27432936429977417, 0.2724132239818573, 0.27346503734588623, 0.2758690118789673, 0.27701064944267273, 0.27279841899871826, 0.2717286944389343, 0.27150872349739075, 0.27118727564811707, 0.2714857757091522, 0.27258923649787903, 0.2721998691558838, 0.27041882276535034, 0.2701268196105957, 0.2711337208747864, 0.2728468179702759, 0.2671411633491516, 0.27339115738868713, 0.2734895646572113, 0.2681798040866852, 0.274287611246109, 0.2704416811466217]} 2021-12-01 23:29:42,367 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 28.0 2021-12-01 23:29:42,701 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 28.0 2021-12-01 23:29:43,059 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 28.0 2021-12-01 23:29:43,059 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 28.0, '172.23.110.66': 28.0, '172.23.110.64': 28.0} 2021-12-01 23:29:43,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:29:43,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:43,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:29:43,546 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:43,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:29:43,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:29:43,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:29:43,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:29:43,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:29:43,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:29:43,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:29:43,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 23:29:43,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-01 23:29:43,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 23:29:43,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:43,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:43,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:43,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:43,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:44,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:44,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:44,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:44,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 23:29:44,157 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-01 23:29:44,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 23:29:44,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:44,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 23:29:44,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-01 23:29:44,226 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:29:44,230 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:29:44,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:44,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:29:44,706 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:44,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:29:44,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:29:44,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:29:44,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:29:44,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:29:44,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:29:44,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:29:44,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 23:29:44,858 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-01 23:29:44,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 23:29:44,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:44,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:45,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:45,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:45,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:45,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:45,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:45,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:45,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 23:29:45,198 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-01 23:29:45,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 23:29:45,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:45,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 23:29:45,247 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-01 23:29:45,249 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:29:45,250 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:29:45,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:45,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:29:45,737 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:45,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:29:45,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:29:45,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:29:45,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:29:45,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:29:45,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:29:45,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:29:45,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 23:29:45,897 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-01 23:29:45,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 23:29:45,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:46,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:46,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:46,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:46,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:46,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:46,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:46,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:46,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 23:29:46,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-01 23:29:46,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 23:29:46,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:46,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 23:29:46,242 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-01 23:29:46,242 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 3.1: Update 40000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 6688.0234375 # ############################################################ Read Start: 0 Read End: 0 Update Start: 40000000 Update End: 80000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 40000000 Final End: 80000000 2021-12-01 23:29:46,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:46,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638430187.29 2021-12-01 23:29:47,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:47,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638430188.3 2021-12-01 23:29:48,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:48,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638430189.31 2021-12-01 23:29:49,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:49,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638430190.32 2021-12-01 23:29:50,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:50,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638430191.33 2021-12-01 23:29:51,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:51,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638430192.34 2021-12-01 23:29:52,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:52,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638430193.4 2021-12-01 23:29:53,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:53,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638430194.41 2021-12-01 23:29:54,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:54,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638430195.43 2021-12-01 23:29:55,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:55,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638430196.49 2021-12-01 23:29:56,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:56,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:29:57,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:57,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:29:58,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:58,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:29:59,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:59,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:00,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:00,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:01,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:01,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:02,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:02,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:03,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:03,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:04,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:04,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:05,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:05,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:06,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:06,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:07,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:07,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:08,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:08,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:09,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:09,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:10,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:10,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:11,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:11,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:12,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:12,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:13,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:13,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:14,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:14,775 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:15,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:15,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:16,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:16,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:17,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:17,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:18,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:18,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:19,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:19,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:20,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:20,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:21,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:21,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:22,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:22,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:23,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:23,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:24,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:24,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:25,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:25,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:26,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:26,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:27,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:27,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:28,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:28,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:30,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:30,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:31,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:31,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:32,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:32,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:33,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:33,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:34,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:34,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:35,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:35,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:36,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:36,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection40_2_1638430195.43 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638430197.53 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638430194.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638430198.54 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638430187.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638430199.55 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638430191.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638430200.57 Loader_GleamBookUsers0__default_VolumeCollection44_8_1638430189.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638430201.58 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638430193.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638430202.59 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638430188.3 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638430203.6 Loader_GleamBookUsers0__default_VolumeCollection49_5_1638430192.34 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638430204.64 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638430190.32 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638430205.65 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638430196.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638430206.66 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638430197.53 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638430207.67 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638430199.55 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638430208.69 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638430198.54 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638430209.7 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638430200.57 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638430210.71 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638430201.58 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638430211.72 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638430202.59 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638430212.74 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638430203.6 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638430213.75 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638430204.64 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638430214.76 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638430205.65 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638430215.78 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638430206.66 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638430216.79 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638430207.67 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638430217.8 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638430208.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638430218.81 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638430209.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638430219.83 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638430210.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638430220.84 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638430211.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638430221.85 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638430212.74 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638430222.87 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638430213.75 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638430223.92 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638430214.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638430224.93 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638430215.78 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638430225.95 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638430216.79 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638430226.96 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638430217.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638430227.97 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638430219.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638430228.98 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638430218.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638430230.0 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638430220.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638430231.06 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638430221.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638430232.07 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638430223.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638430233.08 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638430222.87 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638430234.09 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638430224.93 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638430235.11 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638430225.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638430236.12 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638430226.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638430237.13 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638430227.97 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638430230.0 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638430232.07 is completed! Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638430228.98 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638430231.06 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638430236.12 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638430233.08 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638430235.11 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638430234.09 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638430237.13 is completed! 2021-12-02 03:13:46,660 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-02 03:13:46,664 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:13:46,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:47,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:13:47,119 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:47,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:13:47,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:13:47,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:13:47,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:13:47,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:13:47,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:13:47,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:13:47,243 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-02 03:13:47,244 | infra | INFO | pool-3-thread-17 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24' started 2021-12-02 03:13:47,244 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24 2021-12-02 03:13:47,246 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-02 03:13:47,249 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:13:47,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:47,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:13:47,701 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:47,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:13:47,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:13:47,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:13:47,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:13:47,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:13:47,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:13:47,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:13:47,846 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-02 03:13:47,848 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85 2021-12-02 03:13:47,848 | infra | INFO | pool-3-thread-18 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85' started 2021-12-02 03:13:47,849 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-02 03:13:47,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:13:47,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:48,078 | test | DEBUG | pool-3-thread-17 | [task:_compare:2725] Comparing 0 == 0 2021-12-02 03:13:48,078 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-02 03:13:48,078 | infra | INFO | pool-3-thread-17 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24' completed 2021-12-02 03:13:48,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:13:48,334 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:48,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:13:48,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:13:48,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:13:48,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:13:48,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:13:48,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:13:48,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:13:48,457 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-02 03:13:48,457 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46 2021-12-02 03:13:48,459 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24 2021-12-02 03:13:48,459 | infra | INFO | pool-3-thread-19 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46' started 2021-12-02 03:13:48,460 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:13:48,460 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-02 03:13:48,460 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85 2021-12-02 03:13:48,625 | test | DEBUG | pool-3-thread-18 | [task:_compare:2725] Comparing 0 == 0 2021-12-02 03:13:48,625 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-02 03:13:48,627 | infra | INFO | pool-3-thread-18 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85' completed 2021-12-02 03:13:48,627 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:13:48,628 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46 2021-12-02 03:13:49,240 | test | DEBUG | pool-3-thread-19 | [task:_compare:2725] Comparing 0 == 0 2021-12-02 03:13:49,240 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-02 03:13:49,240 | infra | INFO | pool-3-thread-19 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46' completed 2021-12-02 03:13:49,242 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:13:49,279 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 20033, u'diskFetches': 10, u'dataUsed': 7390024885860L, u'itemCount': 4000000000L, u'quotaPercentUsed': 78.80127166459958, u'storageTotals': {u'hdd': {u'usedByData': 13965692595537L, u'used': 14197589708631L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 21778220282025L}, u'ram': {u'usedByData': 253836663512L, u'used': 408488972288L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 13965692595537L, u'vbActiveNumNonResident': 3867227113L, u'memUsed': 253836663512L} 2021-12-02 03:13:49,654 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+-------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+-------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 4000000000 | 300.00 GiB | 236.40 GiB | 12.70 TiB | 3.319322175 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+-------------+ 2021-12-02 03:13:49,654 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+-------------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+-------------------+---------------+---------------+---------------+ | 4000000000 | 4000000000 | 40000000-80000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+-------------------+---------------+---------------+---------------+ 2021-12-02 03:13:50,000 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.319322175 2021-12-02 03:13:50,003 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:13:50,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:50,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:13:50,457 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:50,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:13:50,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:13:50,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:13:50,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:13:50,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:13:50,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:13:50,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:13:50,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 03:13:53,892 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 302 2021-12-02 03:13:53,892 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 309 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-731 is 343 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 312 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 302 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 338 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-14 is 350 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-87 is 359 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 325 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-304 is 302 2021-12-02 03:13:53,895 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-305 is 313 2021-12-02 03:13:53,895 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-161 is 310 2021-12-02 03:13:53,895 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-89 is 309 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-362 is 326 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-290 is 302 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-310 is 314 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 311 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 323 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 303 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 312 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 316 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 300 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 322 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 367 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 329 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 342 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-321 is 310 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 352 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 316 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 321 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-148 is 346 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 309 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 318 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-261 is 339 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-725 is 372 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 304 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 300 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-122 is 358 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-268 is 312 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-124 is 312 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-847 is 315 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 301 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 307 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-275 is 331 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-134 is 311 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-281 is 350 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 303 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 322 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 324 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 365 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-141 is 328 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-151 is 322 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 339 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 335 2021-12-02 03:13:53,904 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:13:53,907 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:13:54,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:54,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:13:54,351 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:54,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:13:54,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:13:54,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:13:54,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:13:54,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:13:54,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:13:54,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:13:54,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 03:13:54,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 03:13:55,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 03:13:55,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 03:13:55,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 03:13:57,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-432 is 301 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 349 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-155 is 310 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 376 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-159 is 313 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-953 is 300 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-17 is 317 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-667 is 320 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-650 is 318 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-20 is 306 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-532 is 305 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-389 is 302 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-462 is 319 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-31 is 302 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-609 is 310 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-471 is 375 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-978 is 306 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-547 is 300 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-44 is 312 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-477 is 319 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 314 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 301 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-984 is 323 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-409 is 321 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 302 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-556 is 326 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 307 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-52 is 350 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-414 is 301 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 305 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-919 is 310 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-487 is 313 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-489 is 301 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-922 is 315 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-419 is 326 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-131 is 345 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-78 is 337 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-925 is 327 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-428 is 307 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-68 is 304 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-141 is 331 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-441 is 358 2021-12-02 03:13:57,730 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:13:57,733 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:13:57,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:58,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:13:58,196 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:58,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:13:58,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:13:58,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:13:58,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:13:58,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:13:58,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:13:58,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:13:58,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 03:13:58,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 03:13:58,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 03:13:59,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 03:13:59,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 03:14:01,446 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-289 is 304 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-300 is 319 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-517 is 303 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-230 is 370 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-1023 is 323 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-218 is 302 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-668 is 318 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-670 is 325 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-310 is 400 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-963 is 305 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-753 is 311 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-322 is 305 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-683 is 302 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-179 is 332 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-686 is 303 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-182 is 306 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-724 is 302 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-764 is 304 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-981 is 321 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-550 is 323 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-334 is 306 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-913 is 302 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-553 is 305 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-988 is 314 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-921 is 333 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-929 is 304 2021-12-02 03:14:01,451 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-656 is 337 2021-12-02 03:14:01,451 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-225 is 363 2021-12-02 03:14:01,451 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:14:01,451 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 13352424MB 2021-12-02 03:14:01,451 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 6403MB 2021-12-02 03:14:01,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 379303MB 2021-12-02 03:14:01,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 12972713MB 2021-12-02 03:14:01,456 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:14:01,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:14:01,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:14:01,900 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:14:01,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:14:01,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:14:01,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:14:01,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:14:01,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:14:02,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:14:02,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:14:02,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 03:14:02,121 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-02 03:14:02,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 03:14:02,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 03:14:03,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 03:14:03,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 03:14:04,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 03:14:04,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 03:14:04,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 03:14:05,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 03:14:05,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 03:14:06,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 03:14:06,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 03:14:07,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 03:14:07,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 03:14:08,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 03:14:08,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 03:14:08,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 03:14:09,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 03:14:09,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 03:14:10,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 03:14:10,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 03:14:10,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 03:14:11,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 03:14:11,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 03:14:12,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 03:14:12,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 03:14:13,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 03:14:13,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 03:14:14,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 03:14:14,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 03:14:15,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 03:14:15,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 03:14:16,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 03:14:16,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 03:14:16,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 03:14:17,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 03:14:17,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 03:14:18,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 03:14:18,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 03:14:19,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 03:14:19,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 03:14:20,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 03:14:20,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 03:14:21,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 03:14:21,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 03:14:21,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 03:14:22,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 03:14:22,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 03:14:23,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 03:14:23,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 03:14:24,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 03:14:24,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 03:14:24,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 03:14:25,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 03:14:25,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 03:14:26,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 03:14:26,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 03:14:27,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 03:14:27,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 03:14:28,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 03:14:28,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 03:14:29,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 03:14:29,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 03:14:30,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 03:14:30,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 03:14:30,816 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:14:30,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:14:30,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:14:31,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:14:31,289 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:14:31,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:14:31,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:14:31,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:14:31,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:14:31,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:14:31,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:14:31,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:14:31,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 03:14:31,532 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-02 03:14:31,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 03:14:32,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 03:14:32,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 03:14:32,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 03:14:33,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 03:14:33,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 03:14:34,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 03:14:34,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 03:14:35,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 03:14:35,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 03:14:36,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 03:14:36,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 03:14:37,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 03:14:37,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 03:14:38,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 03:14:38,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 03:14:38,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 03:14:39,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 03:14:39,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 03:14:40,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 03:14:40,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 03:14:41,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 03:14:41,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 03:14:42,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 03:14:42,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 03:14:43,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 03:14:43,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 03:14:43,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 03:14:44,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 03:14:44,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 03:14:45,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 03:14:45,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 03:14:46,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 03:14:46,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 03:14:46,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 03:14:47,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 03:14:47,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 03:14:48,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 03:14:48,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 03:14:49,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 03:14:49,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 03:14:50,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 03:14:50,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 03:14:51,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 03:14:51,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 03:14:52,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 03:14:52,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 03:14:53,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 03:14:53,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 03:14:54,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 03:14:54,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 03:14:55,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 03:14:55,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 03:14:55,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 03:14:56,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 03:14:56,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 03:14:57,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 03:14:57,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 03:14:58,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 03:14:58,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 03:14:59,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 03:14:59,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 03:15:00,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 03:15:00,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 03:15:00,953 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:15:00,956 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:15:01,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:01,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:15:01,433 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:01,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:15:01,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:15:01,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:15:01,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:15:01,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:15:01,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:15:01,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:15:01,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 03:15:01,686 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-02 03:15:01,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 03:15:02,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 03:15:02,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 03:15:03,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 03:15:03,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 03:15:03,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 03:15:04,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 03:15:04,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 03:15:05,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 03:15:05,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 03:15:06,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 03:15:06,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 03:15:07,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 03:15:07,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 03:15:07,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 03:15:08,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 03:15:08,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 03:15:09,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 03:15:09,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 03:15:10,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 03:15:10,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 03:15:11,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 03:15:11,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 03:15:12,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 03:15:12,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 03:15:13,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 03:15:13,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 03:15:14,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 03:15:14,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 03:15:15,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 03:15:15,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 03:15:16,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 03:15:16,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 03:15:17,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 03:15:17,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 03:15:17,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 03:15:18,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 03:15:18,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 03:15:19,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 03:15:19,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 03:15:20,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 03:15:20,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 03:15:21,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 03:15:21,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 03:15:22,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 03:15:22,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 03:15:23,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 03:15:23,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 03:15:24,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 03:15:24,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 03:15:25,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 03:15:25,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 03:15:25,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 03:15:26,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 03:15:26,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 03:15:27,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 03:15:27,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 03:15:28,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 03:15:28,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 03:15:29,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 03:15:29,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 03:15:30,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 03:15:30,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 03:15:31,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 03:15:31,509 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:15:31,509 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 29510.5, u'ReplayWALOffset': u'8625:9383936', u'NTablesCreated': 779953, u'RecentReadIOAmpSet': 0.04051939770579338, u'ReadIOAmp': 1789.48388671875, u'BlockCacheMisses': 55723674, u'NFlushes': 186271, u'RecentWriteAmp': 2.6661572456359863, u'NumSyncFlushes': 1882, u'TotalMemUsed': 132578490, u'NBloomFilterHits': 971995710, u'Fragmentation': 0.4678795337677002, u'NumNormalFlushes': 101732, u'RecentBlockCacheHitRatio': 96.22058868408203, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 419388, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 3187736, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 579341508, u'CheckpointOverhead': 7061504, u'NItemsGet': 0, u'NWriteIOs': 637823, u'NReadBytes': 33171057492L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 20835633, u'SizeRatio': 0.4516001831401478, u'NWriteBytes': 1329532928, u'BloomFilterMemUsed': 0, u'NumTables': 75, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 876544}, {u'TargetSize': 46137344, u'LogicalSize': 21637391, u'SizeRatio': 0.4689778197895397, u'NWriteBytes': 1142763520, u'BloomFilterMemUsed': 0, u'NumTables': 18, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 303104}, {u'TargetSize': 712485495, u'LogicalSize': 599007819, u'SizeRatio': 0.840729843910717, u'NWriteBytes': 4069081088L, u'BloomFilterMemUsed': 0, u'NumTables': 215, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 7024640}, {u'TargetSize': 36270067930L, u'LogicalSize': 34591615752L, u'SizeRatio': 0.953723489538554, u'NWriteBytes': 6061010944L, u'BloomFilterMemUsed': 0, u'NumTables': 783, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 342110208}, {u'TargetSize': 142497100002L, u'LogicalSize': 72540135861L, u'SizeRatio': 0.5090639448801546, u'NWriteBytes': 95814594560L, u'BloomFilterMemUsed': 0, u'NumTables': 253, u'WastedSpace': 1291585860, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 72540135424L}], u'NItemsGarbageCollected': 19275062, u'NInternalFragmentationCompacts': 44, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 444909502, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 150720, u'NNonL0LocalCompacts': 28, u'NWriterCompacts': 0, u'NReadIOs': 11277433, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 442740, u'CompactWriter': {u'NDataBlockWrites': 4378925, u'NWriteBytes': 13050302464L, u'NIndexBlockWrites': 220240, u'NBytesIncoming': 17878319148L, u'NSyncs': 0, u'NInserts': 683924198, u'NWriteIOs': 208882}, u'NWriteBytes': 108319333843L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 19275062, u'LogicalDiskSize': 72540135861L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8696396090740227, u'BufferMemUsed': 502852, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 37948520109L, u'NTableFiles': 1515, u'NFileCountCompacts': 28, u'TableObjectMemUsed': 389360, u'TreeSnapshotMemoryUsed': 181824, u'TotalDiskUsage': 94682193920L, u'BytesIncoming': 124556578050L, u'NItemsDel': 0, u'PendingCheckpointBytes': 3960832, u'NCompacts': 59957, u'NItemsPut': 128654800, u'NSyncs': 84539, u'CompactReader': {u'NReadIOs': 11277433, u'NReadBytes': 33171057492L, u'NDataBlockReads': 10957406, u'NBytesOutgoing': 0, u'NIndexBlockReads': 320027}, u'ActiveDiskUsage': 72890449920L, u'NTablesDeleted': 257254, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 28906525, u'NWriteBytes': 95269031379L, u'NIndexBlockWrites': 694448, u'NBytesIncoming': 125711676935L, u'NSyncs': 84539, u'NInserts': 128651016, u'NWriteIOs': 428941}}, u'NReadBytes': 93176587073593L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 39.56666564941406, u'NInternalFragmentationCompacts': 44, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 471542528, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 8192, u'NTxns': 7170775, u'NWriteBytes': 145858854912L, u'NSegments': 8625, u'NSyncs': 7170775, u'NLoggedBytes': 145858854912L, u'NActiveSegments': 1, u'NWrites': 7194594, u'NInserts': 149859644, u'DiskUsed': 15843328}, u'NGets': 19497058, u'NReadBytesSet': 81874269163L, u'NReadBytesCompact': 95274558710L, u'IndexResidentRatio': 0.03880370780825615, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 72.58677673339844, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 93081312192465L, u'BloomFilterMemUsedRatio': 0.27977877855300903, u'NWriteBytes': 322436812447L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.5339691638946533, u'NBlocksCached': 29252, u'LogicalDataSize': 39523728183L, u'ActiveBloomFilterMemUsed': 41652448, u'NFileCountCompacts': 19044, u'NTableFiles': 2367, u'FlushQueueSize': 72, u'TableObjectMemUsed': 600656, u'BlockCacheQuota': 84914312, u'TotalDiskUsage': 96660115456L, u'NPurgerRuns': 165, u'ReadIOAmpSet': 0.0, u'NSyncs': 253617, u'NReadIOsSetPerSec': 8.633333206176758, u'NReadIOsPerSec': 61.36666488647461, u'NTablesDeleted': 617260, u'BloomFilterFPR': 0.015472772531211376, u'NBloomFilterHitsPerSec': 1704.2332763671875, u'WriteCacheMemUsed': 4402248, u'RecentBloomFilterFPR': 0.01586109958589077, u'BytesPerRead': 4769921.5, u'BytesIncomingPerSec': 389870.28125, u'NReadIOsGetPerSec': 11.366666793823242, u'NGetStats': 454967, u'TotalIndexBlocksSize': 606484151, u'NGetStatsPerSec': 2.9666666984558105, u'NWriteBytesCompact': 73088471040L, u'NReadIOsSet': 30751962, u'ActiveDataSize': 39523728183L, u'NWriteBytesCompactPerSec': 127112.53125, u'NBloomFilterMissesPerSec': 27.46666717529297, u'NCommitBatches': 7170775, u'NGetStatsComputed': 454967, u'NWriteIOs': 8467496, u'NReadIOsGet': 34920423232L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 21228578, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 213.06666564941406, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 232512, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 83599645, u'NWriterCompacts': 0, u'NReadIOs': 34955141501L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1442938347, u'NTablesCreated': 189539, u'ReadIOAmp': 331.7313335804761, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 971995710, u'TotalIndexBlocksSize': 27130403, u'CheckpointOverhead': 35123200, u'NItemsGet': 105267108, u'NWriteIOs': 379514, u'NReadBytes': 93143210197096L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2090929, u'SizeRatio': 0.04531966556202282, u'NWriteBytes': 6725984256L, u'BloomFilterMemUsed': 28224, u'NumTables': 94, u'WastedSpace': 134, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1986560}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 33479246, u'SizeRatio': 0.7256431146101519, u'NWriteBytes': 24813813760L, u'BloomFilterMemUsed': 1039712, u'NumTables': 61, u'WastedSpace': 58963, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 33464320}, {u'TargetSize': 173564364, u'LogicalSize': 160560748, u'SizeRatio': 0.9250789983593637, u'NWriteBytes': 13411282944L, u'BloomFilterMemUsed': 4899840, u'NumTables': 81, u'WastedSpace': 442231, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 160997376}, {u'TargetSize': 1735643687, u'LogicalSize': 1735643687, u'SizeRatio': 1.0, u'NWriteBytes': 22220365824L, u'BloomFilterMemUsed': 35683328, u'NumTables': 232, u'WastedSpace': 6105267, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1741746176}], u'NItemsGarbageCollected': 38125569, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 26628526, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 57792, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 34943497085L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 182953, u'CompactWriter': {u'NDataBlockWrites': 21960841, u'NWriteBytes': 59960369152L, u'NIndexBlockWrites': 282854, u'NBytesIncoming': 83690334126L, u'NSyncs': 0, u'NInserts': 1442938347, u'NWriteIOs': 133117}, u'NWriteBytes': 66814208929L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 34920398119L, u'NReadBytes': 93081276298878L, u'NDataBlockReads': 66261835280L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 130000018657L}, u'NBloomFilterMisses': 15275828, u'NItemsMerged': 0, u'LogicalDiskSize': 1735643687, u'TotalBloomFilterMemUsed': 41722088, u'WriteAmp': 15.89781254050195, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 41651104, u'LogicalDataSize': 1575082939, u'NTableFiles': 602, u'NFileCountCompacts': 9519, u'TableObjectMemUsed': 149296, u'TreeSnapshotMemoryUsed': 95488, u'TotalDiskUsage': 1975873536, u'BytesIncoming': 4202729700L, u'NItemsDel': 0, u'PendingCheckpointBytes': 4878336, u'NCompacts': 12547, u'NItemsPut': 85770050, u'NSyncs': 84539, u'CompactReader': {u'NReadIOs': 23098798, u'NReadBytes': 61933575800L, u'NDataBlockReads': 22662336, u'NBytesOutgoing': 0, u'NIndexBlockReads': 436462}, u'ActiveDiskUsage': 1938194432, u'NTablesDeleted': 189071, u'IterateReader': {u'NReadIOs': 168, u'NReadBytes': 322418, u'NDataBlockReads': 84, u'NBytesOutgoing': 0, u'NIndexBlockReads': 84}, u'BloomFilterFPR': 0.015472772598048906, u'Writer': {u'NDataBlockWrites': 1423750, u'NWriteBytes': 6853839777L, u'NIndexBlockWrites': 162292, u'NBytesIncoming': 4974550414L, u'NSyncs': 84539, u'NInserts': 85768158, u'NWriteIOs': 246397}}, u'NReadBytesCompactPerSec': 126597.46875, u'TableMetaMemUsed': 697433, u'NDeletes': 0, u'BlockCacheHitsPerSec': 1007.3333129882812, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 15275828, u'LogicalDiskSize': 74275904683L, u'NSets': 85770050, u'TotalBloomFilterMemUsed': 41723632, u'BufferMemUsed': 502852, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 158.3333282470703, u'FragmentationWithOverhead': 0.5911061763763428, u'NCompactsPerSec': 0.13333334028720856, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 156107.96875, u'TreeSnapshotMemoryUsed': 322320, u'BytesIncoming': 127245762708L, u'NCompacts': 82001, u'NumPersistentFlushes': 82657, u'NWriteIOsPerSec': 162.93333435058594, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 74829389824L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 2.9666666984558105, u'NReadBytesSetPerSec': 22732.56640625, u'BlockCacheHits': 147549275, u'NWriteBytesPerSec': 1039455.4375, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 171026, u'ReadIOAmp': 0.0017479438881683417, u'MemtablesMemUsed': 1025048, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 12240, u'CheckpointOverhead': 1155072, u'NItemsGet': 14367166, u'NWriteIOs': 255565, u'NReadBytes': 205819005, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 725071, u'SizeRatio': 0.015715490687977184, u'NWriteBytes': 1323290624, u'BloomFilterMemUsed': 640, u'NumTables': 80, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 655360}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125135, u'SizeRatio': 0.002712228081443093, u'NWriteBytes': 77799424, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 42573534, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4500, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 24000, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 366983, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 71740, u'CompactWriter': {u'NDataBlockWrites': 9497, u'NWriteBytes': 77799424, u'NIndexBlockWrites': 9497, u'NBytesIncoming': 43175061, u'NSyncs': 0, u'NInserts': 503341, u'NWriteIOs': 9497}, u'NWriteBytes': 1444414763, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 25113, u'NReadBytes': 35893587, u'NDataBlockReads': 7159646, u'NBytesOutgoing': 0, u'NIndexBlockReads': 7159646}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125135, u'TotalBloomFilterMemUsed': 1544, u'WriteAmp': 0.33780154813157187, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1344, u'LogicalDataSize': 125135, u'NTableFiles': 250, u'NFileCountCompacts': 9497, u'TableObjectMemUsed': 62000, u'TreeSnapshotMemoryUsed': 45008, u'TotalDiskUsage': 2048000, u'BytesIncoming': 4275927008L, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 9497, u'NItemsPut': 42577283, u'NSyncs': 84539, u'CompactReader': {u'NReadIOs': 341870, u'NReadBytes': 169925418, u'NDataBlockReads': 170935, u'NBytesOutgoing': 0, u'NIndexBlockReads': 170935}, u'ActiveDiskUsage': 745472, u'NTablesDeleted': 170935, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 161529, u'NWriteBytes': 1366615339, u'NIndexBlockWrites': 161529, u'NBytesIncoming': 120786928, u'NSyncs': 84539, u'NInserts': 1575979, u'NWriteIOs': 246068}}}}} 2021-12-02 03:15:31,513 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.4678795337677002, 0.4704144597053528, 0.4684290587902069, 0.4738217890262604, 0.46076223254203796, 0.4669061303138733, 0.4657285809516907, 0.46527159214019775, 0.473688542842865, 0.46979084610939026, 0.47239893674850464, 0.4682061970233917, 0.47068697214126587, 0.4619098901748657, 0.46349114179611206, 0.471623957157135, 0.46454066038131714, 0.4611978530883789, 0.46267881989479065, 0.46623316407203674, 0.4673308730125427, 0.4708329439163208, 0.4685091972351074, 0.4638839364051819, 0.46339550614356995, 0.4659776985645294, 0.46676719188690186, 0.4720921516418457, 0.4627762734889984, 0.4611952006816864, 0.4559684097766876, 0.4639139771461487, 0.4596552848815918, 0.45865318179130554, 0.4613683521747589, 0.46210813522338867, 0.46133625507354736, 0.4548838138580322, 0.4564320147037506, 0.4591260850429535, 0.4616096615791321, 0.45816770195961, 0.46023547649383545, 0.45866188406944275, 0.4615785479545593, 0.4585941433906555, 0.4642326533794403, 0.45499342679977417, 0.45828911662101746, 0.4530748426914215, 0.4576333463191986, 0.4608323574066162, 0.4509941041469574, 0.4586050510406494, 0.46922728419303894, 0.45939791202545166, 0.46666958928108215, 0.4600003957748413, 0.45495346188545227, 0.4447706937789917, 0.4628749191761017, 0.46822527050971985, 0.4711731970310211, 0.46755126118659973], '172.23.110.66': [0.46864330768585205, 0.4626341164112091, 0.47323089838027954, 0.4662763476371765, 0.46691837906837463, 0.46995052695274353, 0.45669230818748474, 0.4642087519168854, 0.46639442443847656, 0.4712807536125183, 0.46956440806388855, 0.46940019726753235, 0.4661772549152374, 0.47298118472099304, 0.4672324061393738, 0.47057777643203735, 0.46766337752342224, 0.47070252895355225, 0.46402496099472046, 0.4681248664855957, 0.4663366377353668, 0.47136950492858887, 0.47113168239593506, 0.46848592162132263, 0.4700716435909271, 0.46179670095443726, 0.4727860689163208, 0.4728156626224518, 0.4725644588470459, 0.47081732749938965, 0.4786461591720581, 0.4681923985481262, 0.4748392701148987, 0.46772128343582153, 0.4741034507751465, 0.4774235486984253, 0.4777619540691376, 0.4664112329483032, 0.4755721390247345, 0.4734521806240082, 0.4667011499404907, 0.4697214961051941, 0.4682369828224182, 0.4728599488735199, 0.4679756164550781, 0.4675271511077881, 0.4705016613006592, 0.4722340703010559, 0.46222537755966187, 0.47530847787857056, 0.46649491786956787, 0.4591098725795746, 0.4695936143398285, 0.47504720091819763, 0.4780186116695404, 0.4690927565097809, 0.46506184339523315, 0.4706578850746155, 0.4645273685455322, 0.4712585210800171, 0.4632778763771057, 0.47045019268989563, 0.4673038125038147, 0.4642746150493622], '172.23.110.64': [0.4626557528972626, 0.4712734520435333, 0.4614785611629486, 0.45822304487228394, 0.45755359530448914, 0.4582718312740326, 0.4649643898010254, 0.4602403938770294, 0.46096575260162354, 0.4614753723144531, 0.4683327078819275, 0.4635295271873474, 0.4565392732620239, 0.46480593085289, 0.4623485505580902, 0.46687933802604675, 0.45048174262046814, 0.46289899945259094, 0.4614242911338806, 0.4626357853412628, 0.45815804600715637, 0.46372970938682556, 0.46826374530792236, 0.45391330122947693, 0.45515763759613037, 0.46554455161094666, 0.46476975083351135, 0.467601478099823, 0.4614545404911041, 0.45472997426986694, 0.4645775258541107, 0.46015235781669617, 0.4716290831565857, 0.46474897861480713, 0.46299102902412415, 0.4674052298069, 0.46790823340415955, 0.45965439081192017, 0.46964704990386963, 0.47301429510116577, 0.46939438581466675, 0.4710654020309448, 0.46850088238716125, 0.46558284759521484, 0.4567161500453949, 0.45370209217071533, 0.47107475996017456, 0.47131282091140747, 0.4629392623901367, 0.4658852815628052, 0.4540550112724304, 0.46379509568214417, 0.47148898243904114, 0.4690966308116913, 0.4668698310852051, 0.4587341547012329, 0.4672911763191223, 0.4608316421508789, 0.46135008335113525, 0.46254628896713257, 0.4694972336292267, 0.46748223900794983, 0.4599528908729553, 0.46312496066093445]} 2021-12-02 03:15:31,845 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 47.0 2021-12-02 03:15:32,176 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 47.0 2021-12-02 03:15:32,516 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 47.0 2021-12-02 03:15:32,516 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 47.0, '172.23.110.66': 47.0, '172.23.110.64': 47.0} 2021-12-02 03:15:32,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:15:32,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:32,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:15:32,963 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:32,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:15:32,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:15:33,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:15:33,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:15:33,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:15:33,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:15:33,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:15:33,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:15:33,125 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 03:15:33,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:15:33,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:33,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:33,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:33,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:33,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:33,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:33,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 03:15:33,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 03:15:33,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:15:33,884 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 03:15:33,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:15:33,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:15:33,953 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 03:15:33,953 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:15:33,956 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:15:34,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:34,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:15:34,387 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:34,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:15:34,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:15:34,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:15:34,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:15:34,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:15:34,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:15:34,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:15:34,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:15:34,542 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 03:15:34,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:15:34,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:34,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:34,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:34,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:34,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:34,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:34,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:34,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:34,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:34,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:35,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:35,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:35,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:15:35,073 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 03:15:35,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:15:35,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:35,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:15:35,118 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 03:15:35,118 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:15:35,121 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:15:35,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:35,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:15:35,595 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:35,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:15:35,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:15:35,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:15:35,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:15:35,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:15:35,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:15:35,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:15:35,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:15:35,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 03:15:35,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:15:35,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:35,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:36,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:36,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:36,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:36,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:36,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:36,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:36,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:36,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:36,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:36,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:36,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:15:36,335 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 03:15:36,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:15:36,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:36,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:15:36,378 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 03:15:36,378 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # RAM FootPrint: 7005.04296875 # ############################################################ Read Start: 0 Read End: 40000000 Update Start: 80000000 Update End: 800000000 Expiry Start: 0 Expiry End: 0 Delete Start: 40000000 Delete End: 80000000 Create Start: 80000000 Create End: 800000000 Final Start: 80000000 Final End: 800000000 2021-12-02 03:15:36,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:36,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638443737.43 2021-12-02 03:15:37,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:37,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638443738.44 2021-12-02 03:15:38,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:38,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638443739.45 2021-12-02 03:15:39,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:39,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638443740.46 2021-12-02 03:15:40,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:40,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638443741.47 2021-12-02 03:15:41,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:41,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638443742.48 2021-12-02 03:15:42,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:42,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638443743.51 2021-12-02 03:15:43,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:43,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638443744.52 2021-12-02 03:15:44,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:44,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638443745.53 2021-12-02 03:15:45,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:45,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638443746.54 2021-12-02 03:15:46,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:46,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:47,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:47,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:48,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:48,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:49,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:49,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:50,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:50,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:51,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:51,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:52,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:52,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:53,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:53,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:54,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:54,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:55,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:55,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:56,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:56,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:57,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:57,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:58,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:58,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:59,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:59,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:00,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:00,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:01,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:01,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:02,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:02,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:03,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:03,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:04,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:04,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:05,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:05,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:06,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:06,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:07,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:07,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:08,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:08,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:09,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:09,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:10,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:10,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:11,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:11,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:12,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:12,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:13,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:13,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:14,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:14,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:15,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:15,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:16,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:16,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:17,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:17,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:18,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:18,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:19,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:19,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:20,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:20,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:21,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:21,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:22,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:22,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:23,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:23,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:24,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:24,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:25,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:25,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None ############################################################ # # Step 4: Rebalance in with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.66'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.67', '172.23.110.68'] Servers coming in : ['172.23.110.68'] with services: ['kv'] Servers going out : [] NEW AVAILABLE nodes for cluster: ['172.23.110.67'] 2021-12-02 03:16:26,993 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.68]_OUT=[]_1638443786.97 2021-12-02 03:16:26,993 | infra | INFO | pool-3-thread-20 | [task:start_task:89] Thread 'Rebalance_task_IN=[172.23.110.68]_OUT=[]_1638443786.97' started 2021-12-02 03:16:26,996 | test | DEBUG | pool-3-thread-20 | [rest_client:add_node:1178] Adding remote node 172.23.110.68:8091 to cluster 172.23.110.64:8091 2021-12-02 03:16:27,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L466 2021-12-02 03:16:27,002 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-02 03:16:31,309 | test | DEBUG | pool-3-thread-20 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-02 03:16:31,364 | test | DEBUG | pool-3-thread-20 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 03:16:31,364 | test | INFO | pool-3-thread-20 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 10.5978222887 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 12.2151615251 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 12.8767008349 | Cluster node | | 172.23.110.68 | ['kv'] | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-02 03:16:31,372 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:16:31,372 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:16:31,385 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:16:31,388 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:16:31,388 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:16:41,403 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:16:41,403 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:16:41,418 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:16:41,421 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:16:41,421 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:16:51,434 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:16:51,434 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:16:51,448 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:16:51,451 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:16:51,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:01,466 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:01,466 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:01,486 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:01,489 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:01,489 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:11,516 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:11,516 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:11,536 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:11,539 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:11,539 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:21,553 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:21,555 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:21,572 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:21,575 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:21,575 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:27,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-02 03:17:27,007 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-02 03:17:31,588 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:31,589 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:31,611 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:31,615 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:31,615 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:41,651 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:41,651 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:41,668 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:41,673 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:41,673 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:51,687 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:51,687 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:51,703 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:51,707 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:51,707 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:57,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:57,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:17:57,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:01,720 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.19 % 2021-12-02 03:18:01,720 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.19 2021-12-02 03:18:01,734 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:01,737 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:01,737 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:07,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.39 % 2021-12-02 03:18:07,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:07,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:11,752 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:11,752 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:11,769 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:11,772 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:11,772 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:17,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:17,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:17,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:21,783 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:21,785 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:21,799 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:21,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:21,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:27,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:27,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:27,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:31,813 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:31,813 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:31,828 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:31,829 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:31,829 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:37,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:37,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:37,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:41,842 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:41,842 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:41,858 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:41,859 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:41,859 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:47,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:47,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:47,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:51,871 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:51,871 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:51,888 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:51,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:51,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:57,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:57,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:57,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:01,904 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:01,904 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:19:01,924 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:01,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:01,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:07,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:07,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:07,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:11,940 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:11,940 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:19:11,954 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:11,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:11,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:17,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:17,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:17,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:21,970 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:21,970 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:19:21,992 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:21,993 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:21,993 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:27,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.98 % 2021-12-02 03:19:27,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:27,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:32,006 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.98 % 2021-12-02 03:19:32,007 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.98 2021-12-02 03:19:32,025 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:32,026 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:32,026 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:37,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 03:19:37,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:37,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:42,040 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:42,042 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:19:42,055 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:42,058 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:42,058 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:47,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:47,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:47,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:52,072 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:52,072 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:19:52,092 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:52,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:52,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:57,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:57,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:57,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:02,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:02,108 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:02,127 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:02,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:02,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:07,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:07,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:07,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:12,142 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:12,142 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:12,161 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:12,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:12,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:17,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:17,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:17,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:22,177 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:22,178 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:22,197 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:22,198 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:22,198 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:27,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:27,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:27,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:32,213 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:32,213 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:32,229 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:32,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:32,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:37,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:37,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:37,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:42,249 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:42,249 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:42,269 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:42,272 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:42,272 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:47,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:47,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:47,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:52,289 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:20:52,289 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.76 2021-12-02 03:20:52,302 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:52,305 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:52,305 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:57,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:20:57,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:57,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:02,318 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:21:02,319 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.76 2021-12-02 03:21:02,332 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:02,335 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:02,335 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:07,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:21:07,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:07,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:12,348 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.15 % 2021-12-02 03:21:12,349 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.15 2021-12-02 03:21:12,368 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:12,369 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:12,369 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:17,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:17,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:17,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:22,384 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:22,384 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:22,404 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:22,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:22,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:27,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:27,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:27,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:32,421 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:32,421 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:32,444 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:32,447 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:32,447 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:37,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:37,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:37,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:42,463 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:42,463 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:42,483 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:42,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:42,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:47,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:47,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:47,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:52,499 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:52,500 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:52,522 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:52,529 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:52,529 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:57,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:57,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:57,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:02,546 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:22:02,546 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:22:02,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:02,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:02,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:07,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:22:07,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:07,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:12,605 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:22:12,605 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:22:12,625 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:12,631 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:12,631 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:17,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 03:22:17,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:17,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:22,647 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 03:22:22,647 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.54 2021-12-02 03:22:22,664 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:22,667 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:22,668 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:27,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 03:22:27,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:27,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:32,680 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 03:22:32,681 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.73 2021-12-02 03:22:32,697 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:32,700 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:32,700 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:37,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 03:22:37,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:37,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:42,713 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 03:22:42,714 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.73 2021-12-02 03:22:42,732 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:42,734 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:42,734 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:47,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.93 % 2021-12-02 03:22:47,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:47,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:52,747 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:22:52,747 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:22:52,763 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:52,766 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:52,766 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:57,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:22:57,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:57,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:02,783 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:02,785 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:02,799 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:02,803 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:02,803 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:07,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:07,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:07,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:12,816 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:12,818 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:12,834 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:12,835 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:12,836 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:17,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:17,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:17,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:22,852 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:22,854 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:22,871 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:22,872 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:22,874 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:27,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:27,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:27,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:32,884 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:32,884 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:32,900 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:32,901 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:32,903 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:37,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:37,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:37,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:42,914 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:42,914 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.32 2021-12-02 03:23:42,927 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:42,930 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:42,930 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:47,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:47,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:47,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:52,940 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:52,940 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.32 2021-12-02 03:23:52,954 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:52,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:52,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:57,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:57,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:57,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:02,967 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 03:24:02,969 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.52 2021-12-02 03:24:02,983 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:02,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:02,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:07,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 03:24:07,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:07,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:12,997 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 03:24:12,997 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.52 2021-12-02 03:24:13,012 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:13,013 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:13,013 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:17,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:17,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:17,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:23,026 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:23,026 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:23,052 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:23,055 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:23,055 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:27,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:27,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:27,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:33,068 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:33,068 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:33,082 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:33,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:33,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:37,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:37,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:37,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:43,098 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:43,098 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:43,117 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:43,118 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:43,119 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:47,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:47,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:47,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:53,131 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:53,132 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:53,148 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:53,151 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:53,151 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:57,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:57,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:57,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:03,164 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:25:03,164 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:25:03,178 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:03,180 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:03,180 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:07,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:07,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:07,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:13,194 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:13,194 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.1 2021-12-02 03:25:13,213 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:13,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:13,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:17,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:18,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:18,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:23,229 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:23,230 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.1 2021-12-02 03:25:23,250 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:23,253 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:23,253 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:28,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 03:25:28,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:28,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:33,266 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 03:25:33,266 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.3 2021-12-02 03:25:33,289 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:33,290 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:33,292 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:38,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 03:25:38,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:38,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:43,305 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.49 % 2021-12-02 03:25:43,305 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.49 2021-12-02 03:25:43,325 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:43,326 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:43,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:48,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:25:48,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:48,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:53,341 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:25:53,341 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:25:53,361 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:53,364 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:53,364 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:58,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:25:58,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:58,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:03,377 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:03,378 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:26:03,398 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:03,400 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:03,400 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:08,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:08,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:08,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:13,413 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:13,413 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:26:13,434 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:13,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:13,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:18,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:18,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:18,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:23,453 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:23,453 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:26:23,470 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:23,473 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:23,473 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:28,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:28,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:28,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:33,493 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:33,493 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.88 2021-12-02 03:26:33,513 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:33,517 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:33,517 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:38,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:38,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:38,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:43,532 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:43,532 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.88 2021-12-02 03:26:43,549 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:43,552 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:43,552 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:48,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:48,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:48,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:53,565 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.08 % 2021-12-02 03:26:53,566 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.08 2021-12-02 03:26:53,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:53,589 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:53,589 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:58,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.08 % 2021-12-02 03:26:58,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:58,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:03,605 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.27 % 2021-12-02 03:27:03,605 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.27 2021-12-02 03:27:03,621 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:03,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:03,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:08,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.27 % 2021-12-02 03:27:08,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:08,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:13,638 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.27 % 2021-12-02 03:27:13,638 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.27 2021-12-02 03:27:13,657 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:13,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:13,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:18,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:18,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:18,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:23,674 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:23,674 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:23,688 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:23,691 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:23,691 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:28,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:28,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:28,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:33,706 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:33,706 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:33,723 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:33,726 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:33,726 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:38,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:38,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:38,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:43,743 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:43,743 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:43,759 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:43,762 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:43,762 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:48,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:48,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:48,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:53,775 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:53,776 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:53,793 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:53,798 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:53,798 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:58,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:58,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:58,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:03,812 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:28:03,812 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:28:03,828 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:03,832 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:03,832 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:08,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 03:28:08,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:08,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:13,845 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 03:28:13,846 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.86 2021-12-02 03:28:13,862 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:13,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:13,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:18,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 03:28:18,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:18,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:23,880 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:23,881 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.06 2021-12-02 03:28:23,897 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:23,900 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:23,900 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:28,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:28,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:28,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:33,914 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:33,914 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.06 2021-12-02 03:28:33,930 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:33,933 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:33,933 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:38,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:38,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:38,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:43,946 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:43,946 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 03:28:43,960 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:43,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:43,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:48,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:48,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:48,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:53,986 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:53,986 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 03:28:54,002 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:54,003 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:54,003 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:58,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:58,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:58,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:04,016 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:29:04,016 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 03:29:04,029 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:04,032 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:04,032 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:08,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:29:08,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:08,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:14,042 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:14,042 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 03:29:14,056 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:14,059 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:14,059 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:18,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:18,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:18,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:24,069 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:24,071 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 03:29:24,084 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:24,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:24,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:28,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:28,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:28,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:34,096 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:34,098 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 03:29:34,114 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:34,115 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:34,115 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:38,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 03:29:38,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:38,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:44,127 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 03:29:44,128 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.64 2021-12-02 03:29:44,141 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:44,144 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:44,144 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:48,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 03:29:48,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:48,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:54,154 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:29:54,154 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.84 2021-12-02 03:29:54,168 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:54,171 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:54,171 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:58,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:29:58,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:58,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:04,183 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:30:04,184 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.84 2021-12-02 03:30:04,198 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:04,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:04,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:08,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:30:08,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:08,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:14,213 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:14,213 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 03:30:14,229 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:14,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:14,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:18,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:18,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:18,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:24,243 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:24,243 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 03:30:24,262 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:24,263 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:24,263 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:28,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:28,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:28,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:34,276 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:34,278 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 03:30:34,296 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:34,299 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:34,299 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:38,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:38,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:38,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:44,311 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:44,311 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.23 2021-12-02 03:30:44,325 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:44,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:44,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:48,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:48,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:48,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:54,338 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:54,338 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.23 2021-12-02 03:30:54,352 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:54,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:54,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:58,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:58,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:58,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:04,368 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.42 % 2021-12-02 03:31:04,368 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.42 2021-12-02 03:31:04,382 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:04,385 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:04,385 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:08,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.42 % 2021-12-02 03:31:08,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:08,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:14,400 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.42 % 2021-12-02 03:31:14,400 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.42 2021-12-02 03:31:14,414 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:14,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:14,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:18,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:18,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:18,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:24,427 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:24,427 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.62 2021-12-02 03:31:24,443 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:24,444 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:24,444 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:28,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:28,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:28,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:34,457 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:34,459 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.62 2021-12-02 03:31:34,479 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:34,480 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:34,480 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:38,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:38,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:38,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:44,496 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:44,496 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 03:31:44,519 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:44,520 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:44,520 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:48,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:48,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:48,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:54,535 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:54,535 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 03:31:54,555 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:54,558 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:54,558 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:58,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:31:58,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:58,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:04,571 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:04,571 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:04,591 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:04,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:04,594 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:08,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:08,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:08,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:14,609 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:14,609 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:14,631 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:14,634 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:14,634 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:18,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:18,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:18,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:24,651 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:24,653 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:24,673 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:24,677 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:24,677 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:28,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:28,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:28,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:34,690 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:34,690 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:34,706 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:34,709 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:34,709 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:38,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.20 % 2021-12-02 03:32:38,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:38,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:44,720 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.20 % 2021-12-02 03:32:44,720 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.2 2021-12-02 03:32:44,737 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:44,740 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:44,740 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:49,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.20 % 2021-12-02 03:32:49,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:49,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:54,752 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 03:32:54,752 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.4 2021-12-02 03:32:54,766 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:54,767 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:54,769 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:59,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 03:32:59,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:59,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:04,782 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.60 % 2021-12-02 03:33:04,782 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.6 2021-12-02 03:33:04,796 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:04,799 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:04,799 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:09,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.60 % 2021-12-02 03:33:09,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:09,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:14,811 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:14,811 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:14,834 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:14,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:14,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:19,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:19,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:19,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:24,851 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:24,851 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:24,880 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:24,884 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:24,884 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:29,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:29,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:29,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:34,914 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:34,915 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:34,933 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:34,936 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:34,936 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:39,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:39,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:39,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:44,950 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:44,950 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:44,967 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:44,970 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:44,970 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:49,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:49,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:49,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:54,983 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:54,983 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:55,002 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:55,005 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:55,005 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:59,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:59,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:59,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:05,016 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:05,017 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.99 2021-12-02 03:34:05,033 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:05,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:05,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:09,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:09,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:09,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:15,048 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:15,048 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.99 2021-12-02 03:34:15,065 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:15,068 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:15,068 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:19,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:19,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:19,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:25,079 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:25,079 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.99 2021-12-02 03:34:25,095 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:25,096 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:25,096 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:29,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.18 % 2021-12-02 03:34:29,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:29,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:35,107 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.18 % 2021-12-02 03:34:35,107 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.18 2021-12-02 03:34:35,121 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:35,124 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:35,124 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:39,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:39,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:39,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:45,134 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:45,134 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:34:45,148 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:45,150 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:45,150 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:49,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:49,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:49,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:55,160 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:55,160 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:34:55,174 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:55,176 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:55,176 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:59,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:59,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:59,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:05,186 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:05,186 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:35:05,198 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:05,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:05,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:09,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:09,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:09,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:15,210 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:15,211 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:35:15,224 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:15,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:15,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:19,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:19,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:19,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:25,236 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:25,236 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:35:25,253 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:25,256 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:25,256 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:29,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:29,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:29,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:35,266 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:35,267 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.77 2021-12-02 03:35:35,282 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:35,286 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:35,286 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:39,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:39,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:39,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:45,296 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:45,298 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.77 2021-12-02 03:35:45,313 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:45,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:45,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:49,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:49,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:49,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:55,326 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.96 % 2021-12-02 03:35:55,326 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.96 2021-12-02 03:35:55,342 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:55,344 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:55,344 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:59,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.16 % 2021-12-02 03:35:59,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:59,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:05,357 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.16 % 2021-12-02 03:36:05,358 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.16 2021-12-02 03:36:05,377 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:05,378 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:05,378 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:09,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:09,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:09,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:15,391 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:15,391 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 03:36:15,405 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:15,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:15,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:19,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:19,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:19,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:25,418 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:25,420 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 03:36:25,434 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:25,436 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:25,436 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:29,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:29,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:29,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:35,447 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:35,447 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 03:36:35,464 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:35,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:35,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:39,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:39,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:39,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:45,492 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:45,493 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.55 2021-12-02 03:36:45,513 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:45,515 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:45,516 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:49,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:49,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:49,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:55,528 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:55,529 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.55 2021-12-02 03:36:55,546 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:55,549 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:55,549 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:59,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:59,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:59,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:05,561 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:37:05,562 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.55 2021-12-02 03:37:05,584 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:05,585 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:05,586 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:09,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:37:09,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:09,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:15,604 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:15,605 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 03:37:15,618 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:15,621 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:15,621 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:19,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:19,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:19,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:25,632 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:25,634 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 03:37:25,650 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:25,653 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:25,653 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:29,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:29,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:29,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:35,664 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:35,664 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 03:37:35,683 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:35,686 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:35,686 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:39,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.94 % 2021-12-02 03:37:39,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:39,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:45,717 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:45,717 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.13 2021-12-02 03:37:45,733 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:45,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:45,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:49,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:49,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:49,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:55,749 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:55,750 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.13 2021-12-02 03:37:55,765 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:55,767 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:55,769 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:59,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:59,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:59,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:05,779 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:38:05,779 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.13 2021-12-02 03:38:05,796 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:05,798 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:05,799 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:09,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:38:09,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:09,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:15,809 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:15,809 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:15,825 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:15,828 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:15,828 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:19,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:19,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:19,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:25,838 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:25,838 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:25,854 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:25,857 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:25,857 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:29,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:29,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:29,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:35,868 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:35,868 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:35,890 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:35,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:35,892 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:39,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:39,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:39,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:45,904 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:45,905 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:45,924 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:45,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:45,928 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:49,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:49,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:49,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:55,941 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:55,941 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:55,983 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:55,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:55,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:59,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:59,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:59,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:05,996 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:39:05,996 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:39:06,015 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:06,017 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:06,019 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:09,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:39:09,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:09,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:16,040 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.92 % 2021-12-02 03:39:16,042 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.92 2021-12-02 03:39:16,058 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:16,061 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:16,062 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:19,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.92 % 2021-12-02 03:39:19,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:19,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:26,073 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 03:39:26,073 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.11 2021-12-02 03:39:26,091 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:26,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:26,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:29,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 03:39:29,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:29,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:36,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:36,108 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:39:36,153 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:36,155 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:36,155 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:39,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:39,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:39,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:46,167 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:46,167 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:39:46,181 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:46,184 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:46,184 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:49,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:49,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:49,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:56,196 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:56,196 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:39:56,213 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:56,214 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:56,214 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:59,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:59,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:59,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:06,226 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:40:06,226 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:40:06,240 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:06,243 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:06,243 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:09,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:40:09,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:09,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:16,252 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:16,253 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:16,266 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:16,267 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:16,267 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:19,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:19,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:19,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:26,278 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:26,279 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:26,292 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:26,293 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:26,295 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:29,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:29,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:29,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:36,305 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:36,305 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:36,319 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:36,322 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:36,322 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:39,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:39,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:39,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:46,334 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:46,334 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:46,351 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:46,352 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:46,354 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:49,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:49,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:49,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:56,364 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.70 % 2021-12-02 03:40:56,364 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.7 2021-12-02 03:40:56,378 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:56,380 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:56,380 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:59,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.89 % 2021-12-02 03:40:59,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:59,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:06,390 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:06,390 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.09 2021-12-02 03:41:06,404 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:06,405 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:06,405 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:09,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:09,951 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:09,951 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:16,415 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:16,417 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.09 2021-12-02 03:41:16,430 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:16,433 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:16,433 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:19,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:19,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:19,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:26,444 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:26,444 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.09 2021-12-02 03:41:26,463 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:26,466 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:26,466 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:29,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:30,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:30,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:36,477 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:36,477 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:41:36,493 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:36,496 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:36,496 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:40,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:40,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:40,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:46,510 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:46,510 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:41:46,529 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:46,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:46,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:50,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:50,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:50,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:56,545 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:56,545 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:41:56,561 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:56,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:56,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:00,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:00,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:00,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:06,575 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:06,575 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:42:06,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:06,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:06,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:10,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:10,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:10,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:16,602 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:16,604 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:42:16,625 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:16,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:16,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:20,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:20,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:20,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:26,641 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:26,641 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.48 2021-12-02 03:42:26,661 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:26,664 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:26,665 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:30,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:30,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:30,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:36,677 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:36,677 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.48 2021-12-02 03:42:36,693 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:36,696 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:36,696 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:40,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:40,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:40,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:46,707 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:46,709 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.67 2021-12-02 03:42:46,726 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:46,730 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:46,730 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:50,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:50,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:50,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:56,743 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:56,743 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.67 2021-12-02 03:42:56,757 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:56,760 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:56,760 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:00,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:43:00,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:00,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:06,773 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.87 % 2021-12-02 03:43:06,773 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.87 2021-12-02 03:43:06,792 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:06,796 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:06,796 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:10,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:10,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:10,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:16,808 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:16,808 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 03:43:16,834 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:16,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:16,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:20,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:20,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:20,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:26,849 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:26,851 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 03:43:26,865 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:26,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:26,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:30,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:30,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:30,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:36,878 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:36,880 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 03:43:36,894 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:36,897 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:36,897 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:40,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:40,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:40,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:46,907 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:43:46,907 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.26 2021-12-02 03:43:46,921 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:46,924 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:46,924 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:50,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:43:50,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:50,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:56,934 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:43:56,934 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.26 2021-12-02 03:43:56,950 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:56,953 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:56,953 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:00,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:44:00,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:00,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:06,964 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:06,964 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.46 2021-12-02 03:44:06,979 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:06,982 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:06,982 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:10,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:10,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:10,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:16,993 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:16,993 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.46 2021-12-02 03:44:17,012 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:17,015 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:17,015 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:20,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:20,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:20,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:27,026 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 03:44:27,026 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.65 2021-12-02 03:44:27,043 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:27,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:27,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:30,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 03:44:30,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:30,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:37,062 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 03:44:37,062 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.65 2021-12-02 03:44:37,078 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:37,079 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:37,079 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:40,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:40,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:40,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:47,091 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:47,091 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:44:47,107 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:47,109 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:47,109 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:50,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:50,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:50,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:57,121 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:57,121 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:44:57,154 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:57,158 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:57,158 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:00,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:00,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:00,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:07,170 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:07,170 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:45:07,187 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:07,190 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:07,190 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:10,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:10,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:10,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:17,204 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:17,204 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:45:17,221 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:17,223 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:17,224 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:20,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:20,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:20,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:27,237 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 03:45:27,237 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.04 2021-12-02 03:45:27,255 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:27,259 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:27,259 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:30,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:30,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:30,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:37,282 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:37,282 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.24 2021-12-02 03:45:37,303 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:37,309 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:37,309 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:40,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:40,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:40,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:47,322 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:47,322 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.24 2021-12-02 03:45:47,341 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:47,342 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:47,342 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:50,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:50,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:50,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:57,352 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:57,354 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.24 2021-12-02 03:45:57,371 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:57,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:57,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:00,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:46:00,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:00,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:07,387 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.43 % 2021-12-02 03:46:07,388 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.43 2021-12-02 03:46:07,404 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:07,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:07,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:10,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:10,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:10,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:17,418 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:17,420 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.63 2021-12-02 03:46:17,437 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:17,440 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:17,440 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:20,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:20,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:20,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:27,453 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:27,453 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.63 2021-12-02 03:46:27,467 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:27,470 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:27,470 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:30,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:30,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:30,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:37,480 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:37,480 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.63 2021-12-02 03:46:37,497 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:37,500 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:37,500 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:40,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:40,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:40,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:47,510 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.82 % 2021-12-02 03:46:47,510 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.82 2021-12-02 03:46:47,525 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:47,526 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:47,526 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:50,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:46:50,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:50,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:57,538 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:46:57,539 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.02 2021-12-02 03:46:57,559 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:57,561 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:57,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:00,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:47:00,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:00,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:07,573 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:47:07,573 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.02 2021-12-02 03:47:07,592 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:07,595 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:07,595 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:10,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:47:10,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:10,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:17,608 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:17,608 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 03:47:17,627 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:17,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:17,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:20,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:20,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:20,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:27,641 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:27,642 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 03:47:27,664 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:27,667 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:27,667 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:30,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:30,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:30,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:37,680 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:37,680 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 03:47:37,701 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:37,703 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:37,703 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:40,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:40,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:40,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:47,714 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:47:47,716 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:47:47,750 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:47,753 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:47,753 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:50,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:47:50,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:50,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:57,765 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:47:57,765 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:47:57,782 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:57,785 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:57,786 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:00,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:00,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:00,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:07,796 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:07,798 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:48:07,815 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:07,819 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:07,819 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:10,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:10,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:10,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:17,832 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:17,832 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:48:17,851 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:17,854 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:17,855 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:20,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:20,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:20,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:27,869 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:27,871 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:48:27,887 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:27,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:27,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:30,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:30,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:30,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:37,903 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:37,904 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:48:37,921 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:37,923 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:37,924 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:40,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:40,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:40,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:47,938 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:47,938 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:48:47,954 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:47,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:47,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:50,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:50,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:50,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:57,967 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:57,967 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:48:57,982 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:57,983 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:57,983 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:00,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:00,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:00,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:07,993 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:07,994 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:49:08,009 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:08,010 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:08,012 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:10,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:11,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:11,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:18,022 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:18,022 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:49:18,038 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:18,039 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:18,039 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:21,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:21,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:21,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:28,052 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:28,053 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.19 2021-12-02 03:49:28,082 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:28,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:28,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:31,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:31,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:31,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:38,098 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:38,098 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.19 2021-12-02 03:49:38,114 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:38,117 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:38,117 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:41,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:41,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:41,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:48,128 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:48,128 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.39 2021-12-02 03:49:48,145 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:48,148 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:48,148 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:51,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:51,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:51,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:58,163 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:58,163 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.39 2021-12-02 03:49:58,180 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:58,183 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:58,183 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:01,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:50:01,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:01,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:08,194 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.58 % 2021-12-02 03:50:08,194 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.58 2021-12-02 03:50:08,213 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:08,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:08,217 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:11,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.58 % 2021-12-02 03:50:11,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:11,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:18,234 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.78 % 2021-12-02 03:50:18,236 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.78 2021-12-02 03:50:18,255 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:18,257 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:18,257 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:21,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.78 % 2021-12-02 03:50:21,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:21,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:28,273 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:28,273 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:28,289 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:28,292 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:28,293 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:31,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:31,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:31,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:38,309 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:38,309 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:38,325 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:38,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:38,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:41,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:41,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:41,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:48,346 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:48,346 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:48,368 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:48,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:48,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:51,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:51,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:51,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:58,390 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:58,390 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:58,408 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:58,413 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:58,413 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:01,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:51:01,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:01,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:08,431 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:51:08,431 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:51:08,450 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:08,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:08,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:11,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:11,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:11,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:18,470 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:18,470 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.17 2021-12-02 03:51:18,484 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:18,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:18,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:21,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:21,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:21,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:28,502 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:28,503 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.17 2021-12-02 03:51:28,516 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:28,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:28,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:31,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:31,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:31,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:38,533 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:38,533 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.17 2021-12-02 03:51:38,552 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:38,555 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:38,555 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:41,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:41,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:41,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:48,568 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.36 % 2021-12-02 03:51:48,569 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.36 2021-12-02 03:51:48,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:48,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:48,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:51,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:51:51,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:51,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:58,601 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:51:58,602 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:51:58,615 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:58,618 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:58,618 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:01,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:01,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:01,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:08,630 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:08,631 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:52:08,644 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:08,647 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:08,647 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:11,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:11,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:11,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:18,661 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:18,661 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:52:18,677 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:18,678 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:18,678 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:21,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:21,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:21,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:28,693 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:28,693 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:52:28,709 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:28,711 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:28,711 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:31,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:31,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:31,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:38,726 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:38,726 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.95 2021-12-02 03:52:38,742 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:38,744 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:38,744 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:41,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:41,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:41,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:48,759 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:48,759 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.95 2021-12-02 03:52:48,778 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:48,779 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:48,779 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:51,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:51,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:51,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:58,792 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:58,792 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.95 2021-12-02 03:52:58,809 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:58,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:58,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:01,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:53:01,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:01,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:08,825 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:08,825 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 03:53:08,839 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:08,842 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:08,842 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:11,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:11,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:11,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:18,859 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:18,861 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 03:53:18,875 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:18,878 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:18,878 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:21,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:21,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:21,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:28,894 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:28,894 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 03:53:28,910 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:28,913 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:28,913 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:31,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:31,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:31,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:38,927 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:38,927 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.54 2021-12-02 03:53:38,944 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:38,947 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:38,947 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:41,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:41,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:41,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:48,960 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:48,961 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.54 2021-12-02 03:53:48,976 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:48,979 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:48,980 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:51,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:51,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:51,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:58,992 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:58,993 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.54 2021-12-02 03:53:59,007 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:59,009 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:59,010 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:01,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:54:01,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:01,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:09,023 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:09,023 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.73 2021-12-02 03:54:09,038 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:09,040 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:09,040 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:11,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:11,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:11,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:19,053 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:19,055 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.73 2021-12-02 03:54:19,072 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:19,073 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:19,073 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:21,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:21,757 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:21,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:29,085 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:29,086 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.73 2021-12-02 03:54:29,101 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:29,104 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:29,104 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:31,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:31,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:31,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:39,118 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.93 % 2021-12-02 03:54:39,118 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.93 2021-12-02 03:54:39,131 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:39,132 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:39,134 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:41,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.93 % 2021-12-02 03:54:41,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:41,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:49,147 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.12 % 2021-12-02 03:54:49,147 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 20.12 2021-12-02 03:54:49,161 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:49,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:49,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:51,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.12 % 2021-12-02 03:54:51,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:51,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:59,176 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.32 % 2021-12-02 03:54:59,176 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 20.32 2021-12-02 03:54:59,190 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:59,191 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:59,191 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:55:01,832 | test | INFO | MainThread | [rest_client:rebalance_reached:119] Rebalance reached >20.12% in 2224.80799985 seconds 2021-12-02 03:55:01,845 | test | DEBUG | MainThread | [rest_client:diag_eval:1453] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-02 03:55:01,845 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-02 03:55:01,848 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:55:01,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:02,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:55:02,237 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:02,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:55:02,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:55:02,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:55:02,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:55:02,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:55:02,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:55:02,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:55:02,371 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 03:55:02,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 03:55:02,428 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['204510 memcached\n'], []) 2021-12-02 03:55:02,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 03:55:02,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 03:55:02,520 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['204510 memcached\n'], []) 2021-12-02 03:55:02,522 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:55:02,526 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:55:02,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:02,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:55:02,868 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:02,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:55:02,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:55:02,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:55:02,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:55:02,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:55:02,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:55:02,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:55:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:55:02,967 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 03:55:03,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:55:03,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:03,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:03,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:03,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:03,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:03,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:03,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:03,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:03,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep